Note: This is test shard 8 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from TableRangeRebalancingTest
[ RUN ] TableRangeRebalancingTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 01:52:27.667200 32391 test_util.cc:268] Using random seed: -621405484
I20240502 01:52:27.680233 32391 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:52:27.680361 32391 ts_itest-base.cc:113] --------------
I20240502 01:52:27.680464 32391 ts_itest-base.cc:114] 3 tablet servers
I20240502 01:52:27.680521 32391 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:52:27.680586 32391 ts_itest-base.cc:116] --------------
2024-05-02T01:52:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:52:27Z Disabled control of system clock
I20240502 01:52:27.705482 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.161.254:41095
--webserver_interface=127.31.161.254
--webserver_port=0
--builtin_ntp_servers=127.31.161.212:34311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.161.254:41095
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 01:52:28.059123 32404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:28.059410 32404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:28.059511 32404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:28.067394 32404 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:28.067514 32404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:52:28.067589 32404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:28.067668 32404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:52:28.067725 32404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:52:28.077270 32404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:34311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.31.161.254:41095
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.161.254:41095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.31.161.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:28.078560 32404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:28.080325 32404 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:28.090797 32409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:28.091030 32410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:28.091408 32412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:28.091624 32404 server_base.cc:1034] running on GCE node
I20240502 01:52:28.092351 32404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:28.093762 32404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:28.094964 32404 hybrid_clock.cc:648] HybridClock initialized: now 1714614748094909 us; error 49 us; skew 500 ppm
I20240502 01:52:28.095311 32404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:28.097476 32404 webserver.cc:460] Webserver started at http://127.31.161.254:36957/ using document root <none> and password file <none>
I20240502 01:52:28.098158 32404 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:28.098294 32404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:28.098590 32404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:28.101271 32404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3fe0b1323a6f45218cf1a7e95403d277"
format_stamp: "Formatted at 2024-05-02 01:52:28 on dist-test-slave-06l8"
I20240502 01:52:28.101980 32404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3fe0b1323a6f45218cf1a7e95403d277"
format_stamp: "Formatted at 2024-05-02 01:52:28 on dist-test-slave-06l8"
I20240502 01:52:28.107417 32404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240502 01:52:28.110656 32418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:28.111907 32404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 01:52:28.112087 32404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3fe0b1323a6f45218cf1a7e95403d277"
format_stamp: "Formatted at 2024-05-02 01:52:28 on dist-test-slave-06l8"
I20240502 01:52:28.112326 32404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:28.131140 32404 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:28.131901 32404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:28.132164 32404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:28.150133 32404 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.254:41095
I20240502 01:52:28.150141 32470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.254:41095 every 8 connection(s)
I20240502 01:52:28.151450 32404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:52:28.155076 32471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:28.161949 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 32404
I20240502 01:52:28.162240 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:52:28.165705 32471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277: Bootstrap starting.
I20240502 01:52:28.170341 32471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:28.171581 32471 log.cc:824] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:28.175709 32471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277: No bootstrap required, opened a new log
I20240502 01:52:28.184726 32471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe0b1323a6f45218cf1a7e95403d277" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 41095 } }
I20240502 01:52:28.185048 32471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:28.185178 32471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fe0b1323a6f45218cf1a7e95403d277, State: Initialized, Role: FOLLOWER
I20240502 01:52:28.185755 32471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe0b1323a6f45218cf1a7e95403d277" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 41095 } }
I20240502 01:52:28.186014 32471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:28.186177 32471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:28.186368 32471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:28.189163 32471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe0b1323a6f45218cf1a7e95403d277" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 41095 } }
I20240502 01:52:28.189754 32471 leader_election.cc:302] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fe0b1323a6f45218cf1a7e95403d277; no voters:
I20240502 01:52:28.190374 32471 leader_election.cc:288] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:28.190523 32476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:28.191610 32476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277 [term 1 LEADER]: Becoming Leader. State: Replica: 3fe0b1323a6f45218cf1a7e95403d277, State: Running, Role: LEADER
I20240502 01:52:28.192229 32476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe0b1323a6f45218cf1a7e95403d277" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 41095 } }
I20240502 01:52:28.192574 32471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:52:28.195305 32477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fe0b1323a6f45218cf1a7e95403d277" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe0b1323a6f45218cf1a7e95403d277" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 41095 } } }
I20240502 01:52:28.195600 32477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:28.196193 32478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fe0b1323a6f45218cf1a7e95403d277. Latest consensus state: current_term: 1 leader_uuid: "3fe0b1323a6f45218cf1a7e95403d277" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe0b1323a6f45218cf1a7e95403d277" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 41095 } } }
I20240502 01:52:28.196417 32478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:28.204771 32484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:52:28.209676 32484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:52:28.217617 32484 catalog_manager.cc:1347] Generated new cluster ID: 5171128074d74424bf9f7f1ff75752d4
I20240502 01:52:28.217722 32484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:52:28.229540 32484 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:52:28.231223 32484 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:52:28.248031 32484 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3fe0b1323a6f45218cf1a7e95403d277: Generated new TSK 0
I20240502 01:52:28.249017 32484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:52:28.256291 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.193:0
--local_ip_for_outbound_sockets=127.31.161.193
--webserver_interface=127.31.161.193
--webserver_port=0
--tserver_master_addrs=127.31.161.254:41095
--builtin_ntp_servers=127.31.161.212:34311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:52:28.584949 32496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:28.585264 32496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:28.585372 32496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:28.592927 32496 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:28.593040 32496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:28.593209 32496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.193
I20240502 01:52:28.602391 32496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:34311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.31.161.193
--webserver_port=0
--tserver_master_addrs=127.31.161.254:41095
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.193
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:28.603717 32496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:28.605486 32496 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:28.617414 32502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:28.617492 32501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:28.617753 32504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:28.618551 32496 server_base.cc:1034] running on GCE node
I20240502 01:52:28.619071 32496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:28.619908 32496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:28.621127 32496 hybrid_clock.cc:648] HybridClock initialized: now 1714614748621054 us; error 37 us; skew 500 ppm
I20240502 01:52:28.621444 32496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:28.623514 32496 webserver.cc:460] Webserver started at http://127.31.161.193:44625/ using document root <none> and password file <none>
I20240502 01:52:28.624183 32496 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:28.624294 32496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:28.624583 32496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:28.627043 32496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "79b0c49d96854b1a8f43e149053ca325"
format_stamp: "Formatted at 2024-05-02 01:52:28 on dist-test-slave-06l8"
I20240502 01:52:28.627882 32496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "79b0c49d96854b1a8f43e149053ca325"
format_stamp: "Formatted at 2024-05-02 01:52:28 on dist-test-slave-06l8"
I20240502 01:52:28.632910 32496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 01:52:28.636202 32510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:28.637481 32496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:52:28.637629 32496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "79b0c49d96854b1a8f43e149053ca325"
format_stamp: "Formatted at 2024-05-02 01:52:28 on dist-test-slave-06l8"
I20240502 01:52:28.637807 32496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:28.651432 32496 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:28.652184 32496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:28.652433 32496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:28.653307 32496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:28.654763 32496 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:28.654857 32496 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:28.654969 32496 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:28.655036 32496 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:28.683904 32496 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.193:35847
I20240502 01:52:28.683939 32623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.193:35847 every 8 connection(s)
I20240502 01:52:28.685202 32496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:52:28.691924 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 32496
I20240502 01:52:28.692121 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:52:28.695626 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.194:0
--local_ip_for_outbound_sockets=127.31.161.194
--webserver_interface=127.31.161.194
--webserver_port=0
--tserver_master_addrs=127.31.161.254:41095
--builtin_ntp_servers=127.31.161.212:34311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:28.698062 32624 heartbeater.cc:347] Connected to a master server at 127.31.161.254:41095
I20240502 01:52:28.698331 32624 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:28.699107 32624 heartbeater.cc:511] Master 127.31.161.254:41095 requested a full tablet report, sending...
I20240502 01:52:28.701088 32435 ts_manager.cc:194] Registered new tserver with Master: 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847)
I20240502 01:52:28.702692 32435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.193:41847
W20240502 01:52:29.013530 32627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:29.013803 32627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:29.013928 32627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:29.021203 32627 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:29.021322 32627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:29.021469 32627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.194
I20240502 01:52:29.030086 32627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:34311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.31.161.194
--webserver_port=0
--tserver_master_addrs=127.31.161.254:41095
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.194
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:29.031284 32627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:29.032920 32627 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:29.043959 32632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:29.044277 32635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:29.044577 32633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:29.044781 32627 server_base.cc:1034] running on GCE node
I20240502 01:52:29.045360 32627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:29.046165 32627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:29.047356 32627 hybrid_clock.cc:648] HybridClock initialized: now 1714614749047312 us; error 34 us; skew 500 ppm
I20240502 01:52:29.047643 32627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:29.049651 32627 webserver.cc:460] Webserver started at http://127.31.161.194:33145/ using document root <none> and password file <none>
I20240502 01:52:29.050297 32627 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:29.050405 32627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:29.050700 32627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:29.053097 32627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "50fc42d44d3441eaa2616635544066a5"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-06l8"
I20240502 01:52:29.053812 32627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "50fc42d44d3441eaa2616635544066a5"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-06l8"
I20240502 01:52:29.058378 32627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240502 01:52:29.061403 32641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:29.062593 32627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20240502 01:52:29.062741 32627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "50fc42d44d3441eaa2616635544066a5"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-06l8"
I20240502 01:52:29.062918 32627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:29.108584 32627 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:29.109472 32627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:29.109741 32627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:29.110648 32627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:29.112088 32627 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:29.112185 32627 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:29.112293 32627 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:29.112361 32627 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:29.145023 32627 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.194:42619
I20240502 01:52:29.145109 32754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.194:42619 every 8 connection(s)
I20240502 01:52:29.146287 32627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:52:29.150833 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 32627
I20240502 01:52:29.151041 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:52:29.154587 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.195:0
--local_ip_for_outbound_sockets=127.31.161.195
--webserver_interface=127.31.161.195
--webserver_port=0
--tserver_master_addrs=127.31.161.254:41095
--builtin_ntp_servers=127.31.161.212:34311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:29.159258 32755 heartbeater.cc:347] Connected to a master server at 127.31.161.254:41095
I20240502 01:52:29.159552 32755 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:29.160526 32755 heartbeater.cc:511] Master 127.31.161.254:41095 requested a full tablet report, sending...
I20240502 01:52:29.162074 32435 ts_manager.cc:194] Registered new tserver with Master: 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619)
I20240502 01:52:29.163084 32435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.194:57811
W20240502 01:52:29.470273 32758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:29.470556 32758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:29.470685 32758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:29.478338 32758 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:29.478505 32758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:29.478648 32758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.195
I20240502 01:52:29.487293 32758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:34311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.31.161.195
--webserver_port=0
--tserver_master_addrs=127.31.161.254:41095
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.195
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:29.488516 32758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:29.490236 32758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:29.501921 32763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:29.502065 32766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:29.502439 32764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:29.503065 32758 server_base.cc:1034] running on GCE node
I20240502 01:52:29.503660 32758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:29.504565 32758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:29.505746 32758 hybrid_clock.cc:648] HybridClock initialized: now 1714614749505695 us; error 41 us; skew 500 ppm
I20240502 01:52:29.506037 32758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:29.508062 32758 webserver.cc:460] Webserver started at http://127.31.161.195:42135/ using document root <none> and password file <none>
I20240502 01:52:29.508772 32758 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:29.508882 32758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:29.509228 32758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:29.511659 32758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "968c06ee65964aceb131114dfbd683de"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-06l8"
I20240502 01:52:29.512369 32758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "968c06ee65964aceb131114dfbd683de"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-06l8"
I20240502 01:52:29.517196 32758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240502 01:52:29.520476 304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:29.521809 32758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:52:29.521947 32758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "968c06ee65964aceb131114dfbd683de"
format_stamp: "Formatted at 2024-05-02 01:52:29 on dist-test-slave-06l8"
I20240502 01:52:29.522130 32758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:29.561306 32758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:29.562179 32758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:29.562445 32758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:29.563398 32758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:29.564942 32758 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:29.565030 32758 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:29.565157 32758 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:29.565229 32758 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:29.596324 32758 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.195:38451
I20240502 01:52:29.596385 417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.195:38451 every 8 connection(s)
I20240502 01:52:29.597625 32758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:52:29.601713 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 32758
I20240502 01:52:29.601910 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:52:29.609956 418 heartbeater.cc:347] Connected to a master server at 127.31.161.254:41095
I20240502 01:52:29.610239 418 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:29.611027 418 heartbeater.cc:511] Master 127.31.161.254:41095 requested a full tablet report, sending...
I20240502 01:52:29.612279 32435 ts_manager.cc:194] Registered new tserver with Master: 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451)
I20240502 01:52:29.613054 32435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.195:40299
I20240502 01:52:29.618139 32391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 01:52:29.636564 32391 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 01:52:29.705386 32624 heartbeater.cc:503] Master 127.31.161.254:41095 was elected leader, sending a full tablet report...
I20240502 01:52:30.165928 32755 heartbeater.cc:503] Master 127.31.161.254:41095 was elected leader, sending a full tablet report...
I20240502 01:52:30.615435 418 heartbeater.cc:503] Master 127.31.161.254:41095 was elected leader, sending a full tablet report...
I20240502 01:52:33.390715 32435 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55862:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\200\007\001\000\000\000\000\000\006\001\000\000\000\000\000\007\001\000\377\377\377\177"
indirect_data: "<redacted>"""
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 8
seed: 0
}
}
W20240502 01:52:33.397276 32435 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 01:52:33.445806 32547 tablet_service.cc:1467] Processing CreateTablet for tablet 06e288c44ec5454caade0f9fd5b86822 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.447062 32551 tablet_service.cc:1467] Processing CreateTablet for tablet a153b9fa8359422eb28faa8f573edb58 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.447217 32687 tablet_service.cc:1467] Processing CreateTablet for tablet 967284a71df1439aa5b27bc48c957c09 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.447995 32547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e288c44ec5454caade0f9fd5b86822. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.447860 32674 tablet_service.cc:1467] Processing CreateTablet for tablet 79527d85ba284c04ae435ed71a3d2d0c (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.448048 32684 tablet_service.cc:1467] Processing CreateTablet for tablet 626b95f7b22d4c278afda157cf50f7cc (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.446225 32553 tablet_service.cc:1467] Processing CreateTablet for tablet 626b95f7b22d4c278afda157cf50f7cc (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.448787 32553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 626b95f7b22d4c278afda157cf50f7cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.448679 32682 tablet_service.cc:1467] Processing CreateTablet for tablet a153b9fa8359422eb28faa8f573edb58 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.448736 32546 tablet_service.cc:1467] Processing CreateTablet for tablet 9e11386999d543a99ca5b08059e3048f (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.449290 32680 tablet_service.cc:1467] Processing CreateTablet for tablet cf049ada294349e3b3811e1397d4851d (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.449368 32555 tablet_service.cc:1467] Processing CreateTablet for tablet f95e1c9c66a44807a89302c6c53d90f2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.449602 32545 tablet_service.cc:1467] Processing CreateTablet for tablet fd7160232773423590368d00a1cb8e88 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.449878 32685 tablet_service.cc:1467] Processing CreateTablet for tablet d1a537af86f146aeb6597e1d61274cf0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.450299 32676 tablet_service.cc:1467] Processing CreateTablet for tablet fd7160232773423590368d00a1cb8e88 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.450387 32543 tablet_service.cc:1467] Processing CreateTablet for tablet 79527d85ba284c04ae435ed71a3d2d0c (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.446971 32686 tablet_service.cc:1467] Processing CreateTablet for tablet f95e1c9c66a44807a89302c6c53d90f2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.450623 32544 tablet_service.cc:1467] Processing CreateTablet for tablet 10898226a6644583915e21ce167afbed (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.451213 32675 tablet_service.cc:1467] Processing CreateTablet for tablet 10898226a6644583915e21ce167afbed (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.451728 32557 tablet_service.cc:1467] Processing CreateTablet for tablet b9f486c135914f558f3c847ac02dda8a (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.451854 32677 tablet_service.cc:1467] Processing CreateTablet for tablet 9e11386999d543a99ca5b08059e3048f (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.452144 32683 tablet_service.cc:1467] Processing CreateTablet for tablet ba40300b09fa4a7f9d0b1b3eaf238ebc (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.452347 32549 tablet_service.cc:1467] Processing CreateTablet for tablet cf049ada294349e3b3811e1397d4851d (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.452735 32679 tablet_service.cc:1467] Processing CreateTablet for tablet 91a40253e73f40089885747aa6c4c635 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.453385 32681 tablet_service.cc:1467] Processing CreateTablet for tablet 29f468ee49544781b84c22fb48f57ede (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.449883 32678 tablet_service.cc:1467] Processing CreateTablet for tablet 06e288c44ec5454caade0f9fd5b86822 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.453800 32688 tablet_service.cc:1467] Processing CreateTablet for tablet b9f486c135914f558f3c847ac02dda8a (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.446633 32548 tablet_service.cc:1467] Processing CreateTablet for tablet 91a40253e73f40089885747aa6c4c635 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.449389 32546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e11386999d543a99ca5b08059e3048f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.453066 32556 tablet_service.cc:1467] Processing CreateTablet for tablet 967284a71df1439aa5b27bc48c957c09 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.454824 32556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 967284a71df1439aa5b27bc48c957c09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.454991 32689 tablet_service.cc:1467] Processing CreateTablet for tablet 657f6ad36cf34cbcaca11cd331672993 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.455293 32548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a40253e73f40089885747aa6c4c635. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.455420 32687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 967284a71df1439aa5b27bc48c957c09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.455968 32549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf049ada294349e3b3811e1397d4851d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.455406 348 tablet_service.cc:1467] Processing CreateTablet for tablet d1a537af86f146aeb6597e1d61274cf0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.457535 348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1a537af86f146aeb6597e1d61274cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.457610 32685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1a537af86f146aeb6597e1d61274cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.455749 32550 tablet_service.cc:1467] Processing CreateTablet for tablet 29f468ee49544781b84c22fb48f57ede (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.458627 32550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f468ee49544781b84c22fb48f57ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.459352 32545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd7160232773423590368d00a1cb8e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.456496 32558 tablet_service.cc:1467] Processing CreateTablet for tablet 657f6ad36cf34cbcaca11cd331672993 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.461354 350 tablet_service.cc:1467] Processing CreateTablet for tablet 967284a71df1439aa5b27bc48c957c09 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.461658 32558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 657f6ad36cf34cbcaca11cd331672993. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.461819 350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 967284a71df1439aa5b27bc48c957c09. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.462527 349 tablet_service.cc:1467] Processing CreateTablet for tablet f95e1c9c66a44807a89302c6c53d90f2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.463291 349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f95e1c9c66a44807a89302c6c53d90f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.464030 352 tablet_service.cc:1467] Processing CreateTablet for tablet 657f6ad36cf34cbcaca11cd331672993 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.464442 352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 657f6ad36cf34cbcaca11cd331672993. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.465204 32543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79527d85ba284c04ae435ed71a3d2d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.465416 346 tablet_service.cc:1467] Processing CreateTablet for tablet ba40300b09fa4a7f9d0b1b3eaf238ebc (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.466621 346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba40300b09fa4a7f9d0b1b3eaf238ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.466722 32557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f486c135914f558f3c847ac02dda8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.456612 32552 tablet_service.cc:1467] Processing CreateTablet for tablet ba40300b09fa4a7f9d0b1b3eaf238ebc (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.467299 32552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba40300b09fa4a7f9d0b1b3eaf238ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.467927 341 tablet_service.cc:1467] Processing CreateTablet for tablet 06e288c44ec5454caade0f9fd5b86822 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.466002 339 tablet_service.cc:1467] Processing CreateTablet for tablet fd7160232773423590368d00a1cb8e88 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.468590 347 tablet_service.cc:1467] Processing CreateTablet for tablet 626b95f7b22d4c278afda157cf50f7cc (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.469457 32551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a153b9fa8359422eb28faa8f573edb58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.469702 32544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10898226a6644583915e21ce167afbed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.469209 340 tablet_service.cc:1467] Processing CreateTablet for tablet 9e11386999d543a99ca5b08059e3048f (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.471844 341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e288c44ec5454caade0f9fd5b86822. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.456703 32554 tablet_service.cc:1467] Processing CreateTablet for tablet d1a537af86f146aeb6597e1d61274cf0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.471689 342 tablet_service.cc:1467] Processing CreateTablet for tablet 91a40253e73f40089885747aa6c4c635 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.469321 338 tablet_service.cc:1467] Processing CreateTablet for tablet 10898226a6644583915e21ce167afbed (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.471158 344 tablet_service.cc:1467] Processing CreateTablet for tablet 29f468ee49544781b84c22fb48f57ede (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 01:52:33.469677 337 tablet_service.cc:1467] Processing CreateTablet for tablet 79527d85ba284c04ae435ed71a3d2d0c (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.465540 351 tablet_service.cc:1467] Processing CreateTablet for tablet b9f486c135914f558f3c847ac02dda8a (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.473906 343 tablet_service.cc:1467] Processing CreateTablet for tablet cf049ada294349e3b3811e1397d4851d (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.474313 32555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f95e1c9c66a44807a89302c6c53d90f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.474308 345 tablet_service.cc:1467] Processing CreateTablet for tablet a153b9fa8359422eb28faa8f573edb58 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e67c484d4c934fb0a2e4040c32f656ca]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 01:52:33.475690 32689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 657f6ad36cf34cbcaca11cd331672993. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.475798 339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd7160232773423590368d00a1cb8e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.476112 340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e11386999d543a99ca5b08059e3048f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.476795 347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 626b95f7b22d4c278afda157cf50f7cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.477043 345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a153b9fa8359422eb28faa8f573edb58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.477516 32678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e288c44ec5454caade0f9fd5b86822. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.477537 337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79527d85ba284c04ae435ed71a3d2d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.477802 338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10898226a6644583915e21ce167afbed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.478104 343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf049ada294349e3b3811e1397d4851d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.478483 32674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79527d85ba284c04ae435ed71a3d2d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.478636 344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f468ee49544781b84c22fb48f57ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.479094 32682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a153b9fa8359422eb28faa8f573edb58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.479187 351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f486c135914f558f3c847ac02dda8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.479640 32680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf049ada294349e3b3811e1397d4851d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.479701 342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a40253e73f40089885747aa6c4c635. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.480331 32684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 626b95f7b22d4c278afda157cf50f7cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.480580 32688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9f486c135914f558f3c847ac02dda8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.482757 32554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1a537af86f146aeb6597e1d61274cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.485942 32683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba40300b09fa4a7f9d0b1b3eaf238ebc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.486220 32676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd7160232773423590368d00a1cb8e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.486769 32686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f95e1c9c66a44807a89302c6c53d90f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.488238 32679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a40253e73f40089885747aa6c4c635. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.489001 32675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10898226a6644583915e21ce167afbed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.489725 32677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e11386999d543a99ca5b08059e3048f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.493170 32681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29f468ee49544781b84c22fb48f57ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:33.553635 437 tablet_bootstrap.cc:492] T 967284a71df1439aa5b27bc48c957c09 P 50fc42d44d3441eaa2616635544066a5: Bootstrap starting.
I20240502 01:52:33.558053 437 tablet_bootstrap.cc:654] T 967284a71df1439aa5b27bc48c957c09 P 50fc42d44d3441eaa2616635544066a5: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.559427 437 log.cc:824] T 967284a71df1439aa5b27bc48c957c09 P 50fc42d44d3441eaa2616635544066a5: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:33.560616 438 tablet_bootstrap.cc:492] T fd7160232773423590368d00a1cb8e88 P 79b0c49d96854b1a8f43e149053ca325: Bootstrap starting.
I20240502 01:52:33.563330 438 tablet_bootstrap.cc:654] T fd7160232773423590368d00a1cb8e88 P 79b0c49d96854b1a8f43e149053ca325: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.564347 438 log.cc:824] T fd7160232773423590368d00a1cb8e88 P 79b0c49d96854b1a8f43e149053ca325: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:33.567021 437 tablet_bootstrap.cc:492] T 967284a71df1439aa5b27bc48c957c09 P 50fc42d44d3441eaa2616635544066a5: No bootstrap required, opened a new log
I20240502 01:52:33.567363 437 ts_tablet_manager.cc:1367] T 967284a71df1439aa5b27bc48c957c09 P 50fc42d44d3441eaa2616635544066a5: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20240502 01:52:33.567363 438 tablet_bootstrap.cc:492] T fd7160232773423590368d00a1cb8e88 P 79b0c49d96854b1a8f43e149053ca325: No bootstrap required, opened a new log
I20240502 01:52:33.568135 438 ts_tablet_manager.cc:1367] T fd7160232773423590368d00a1cb8e88 P 79b0c49d96854b1a8f43e149053ca325: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 01:52:33.571424 439 tablet_bootstrap.cc:492] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de: Bootstrap starting.
I20240502 01:52:33.574896 439 tablet_bootstrap.cc:654] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.576113 439 log.cc:824] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:33.575904 437 raft_consensus.cc:348] T 967284a71df1439aa5b27bc48c957c09 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.576117 438 raft_consensus.cc:348] T fd7160232773423590368d00a1cb8e88 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.576350 437 raft_consensus.cc:374] T 967284a71df1439aa5b27bc48c957c09 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.576468 438 raft_consensus.cc:374] T fd7160232773423590368d00a1cb8e88 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.576447 437 raft_consensus.cc:729] T 967284a71df1439aa5b27bc48c957c09 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fc42d44d3441eaa2616635544066a5, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.576565 438 raft_consensus.cc:729] T fd7160232773423590368d00a1cb8e88 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b0c49d96854b1a8f43e149053ca325, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.577163 437 consensus_queue.cc:260] T 967284a71df1439aa5b27bc48c957c09 P 50fc42d44d3441eaa2616635544066a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.577297 438 consensus_queue.cc:260] T fd7160232773423590368d00a1cb8e88 P 79b0c49d96854b1a8f43e149053ca325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.578752 437 ts_tablet_manager.cc:1398] T 967284a71df1439aa5b27bc48c957c09 P 50fc42d44d3441eaa2616635544066a5: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20240502 01:52:33.579044 439 tablet_bootstrap.cc:492] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de: No bootstrap required, opened a new log
I20240502 01:52:33.579161 437 tablet_bootstrap.cc:492] T d1a537af86f146aeb6597e1d61274cf0 P 50fc42d44d3441eaa2616635544066a5: Bootstrap starting.
I20240502 01:52:33.579285 439 ts_tablet_manager.cc:1367] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240502 01:52:33.580753 438 ts_tablet_manager.cc:1398] T fd7160232773423590368d00a1cb8e88 P 79b0c49d96854b1a8f43e149053ca325: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20240502 01:52:33.581174 438 tablet_bootstrap.cc:492] T 79527d85ba284c04ae435ed71a3d2d0c P 79b0c49d96854b1a8f43e149053ca325: Bootstrap starting.
I20240502 01:52:33.582788 438 tablet_bootstrap.cc:654] T 79527d85ba284c04ae435ed71a3d2d0c P 79b0c49d96854b1a8f43e149053ca325: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.584044 437 tablet_bootstrap.cc:654] T d1a537af86f146aeb6597e1d61274cf0 P 50fc42d44d3441eaa2616635544066a5: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.587719 439 raft_consensus.cc:348] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.588040 439 raft_consensus.cc:374] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.588140 439 raft_consensus.cc:729] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.588708 439 consensus_queue.cc:260] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.589089 437 tablet_bootstrap.cc:492] T d1a537af86f146aeb6597e1d61274cf0 P 50fc42d44d3441eaa2616635544066a5: No bootstrap required, opened a new log
I20240502 01:52:33.589237 437 ts_tablet_manager.cc:1367] T d1a537af86f146aeb6597e1d61274cf0 P 50fc42d44d3441eaa2616635544066a5: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20240502 01:52:33.590167 437 raft_consensus.cc:348] T d1a537af86f146aeb6597e1d61274cf0 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.590373 437 raft_consensus.cc:374] T d1a537af86f146aeb6597e1d61274cf0 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.590469 437 raft_consensus.cc:729] T d1a537af86f146aeb6597e1d61274cf0 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fc42d44d3441eaa2616635544066a5, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.590683 437 consensus_queue.cc:260] T d1a537af86f146aeb6597e1d61274cf0 P 50fc42d44d3441eaa2616635544066a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.591241 439 ts_tablet_manager.cc:1398] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20240502 01:52:33.591272 437 ts_tablet_manager.cc:1398] T d1a537af86f146aeb6597e1d61274cf0 P 50fc42d44d3441eaa2616635544066a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:33.591532 437 tablet_bootstrap.cc:492] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5: Bootstrap starting.
I20240502 01:52:33.591648 439 tablet_bootstrap.cc:492] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de: Bootstrap starting.
I20240502 01:52:33.593251 437 tablet_bootstrap.cc:654] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.593293 439 tablet_bootstrap.cc:654] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.595293 438 tablet_bootstrap.cc:492] T 79527d85ba284c04ae435ed71a3d2d0c P 79b0c49d96854b1a8f43e149053ca325: No bootstrap required, opened a new log
I20240502 01:52:33.595479 438 ts_tablet_manager.cc:1367] T 79527d85ba284c04ae435ed71a3d2d0c P 79b0c49d96854b1a8f43e149053ca325: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20240502 01:52:33.596386 438 raft_consensus.cc:348] T 79527d85ba284c04ae435ed71a3d2d0c P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.596647 438 raft_consensus.cc:374] T 79527d85ba284c04ae435ed71a3d2d0c P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.596689 437 tablet_bootstrap.cc:492] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5: No bootstrap required, opened a new log
I20240502 01:52:33.596805 438 raft_consensus.cc:729] T 79527d85ba284c04ae435ed71a3d2d0c P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b0c49d96854b1a8f43e149053ca325, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.596868 437 ts_tablet_manager.cc:1367] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 01:52:33.597121 438 consensus_queue.cc:260] T 79527d85ba284c04ae435ed71a3d2d0c P 79b0c49d96854b1a8f43e149053ca325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.597757 438 ts_tablet_manager.cc:1398] T 79527d85ba284c04ae435ed71a3d2d0c P 79b0c49d96854b1a8f43e149053ca325: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:33.598071 438 tablet_bootstrap.cc:492] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325: Bootstrap starting.
I20240502 01:52:33.599555 438 tablet_bootstrap.cc:654] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325: Neither blocks nor log segments found. Creating new log.
W20240502 01:52:33.601694 419 tablet.cc:2369] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:52:33.601814 437 raft_consensus.cc:348] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.602023 437 raft_consensus.cc:374] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.602092 437 raft_consensus.cc:729] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fc42d44d3441eaa2616635544066a5, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.602398 437 consensus_queue.cc:260] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.602876 439 tablet_bootstrap.cc:492] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de: No bootstrap required, opened a new log
I20240502 01:52:33.603060 439 ts_tablet_manager.cc:1367] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240502 01:52:33.604139 439 raft_consensus.cc:348] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.604420 439 raft_consensus.cc:374] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.604574 439 raft_consensus.cc:729] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.604947 437 ts_tablet_manager.cc:1398] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 01:52:33.604907 439 consensus_queue.cc:260] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.605636 439 ts_tablet_manager.cc:1398] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:33.605931 439 tablet_bootstrap.cc:492] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de: Bootstrap starting.
I20240502 01:52:33.607699 439 tablet_bootstrap.cc:654] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.609228 437 tablet_bootstrap.cc:492] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5: Bootstrap starting.
I20240502 01:52:33.610420 437 tablet_bootstrap.cc:654] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.610482 438 tablet_bootstrap.cc:492] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325: No bootstrap required, opened a new log
I20240502 01:52:33.610646 438 ts_tablet_manager.cc:1367] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240502 01:52:33.611229 438 raft_consensus.cc:348] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.611465 438 raft_consensus.cc:374] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.611552 438 raft_consensus.cc:729] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b0c49d96854b1a8f43e149053ca325, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.611805 438 consensus_queue.cc:260] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.612346 438 ts_tablet_manager.cc:1398] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:52:33.612638 438 tablet_bootstrap.cc:492] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325: Bootstrap starting.
I20240502 01:52:33.612680 437 tablet_bootstrap.cc:492] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5: No bootstrap required, opened a new log
I20240502 01:52:33.612794 437 ts_tablet_manager.cc:1367] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 01:52:33.613487 437 raft_consensus.cc:348] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } }
I20240502 01:52:33.613670 437 raft_consensus.cc:374] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.613746 437 raft_consensus.cc:729] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fc42d44d3441eaa2616635544066a5, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.614002 437 consensus_queue.cc:260] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } }
I20240502 01:52:33.614172 438 tablet_bootstrap.cc:654] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.614499 437 ts_tablet_manager.cc:1398] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:52:33.614897 439 tablet_bootstrap.cc:492] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de: No bootstrap required, opened a new log
I20240502 01:52:33.615046 439 ts_tablet_manager.cc:1367] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240502 01:52:33.615706 439 raft_consensus.cc:348] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.615927 439 raft_consensus.cc:374] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.616019 439 raft_consensus.cc:729] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.616070 437 tablet_bootstrap.cc:492] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5: Bootstrap starting.
I20240502 01:52:33.616272 439 consensus_queue.cc:260] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.616959 439 ts_tablet_manager.cc:1398] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:52:33.617252 439 tablet_bootstrap.cc:492] T 91a40253e73f40089885747aa6c4c635 P 968c06ee65964aceb131114dfbd683de: Bootstrap starting.
I20240502 01:52:33.618515 437 tablet_bootstrap.cc:654] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.618705 439 tablet_bootstrap.cc:654] T 91a40253e73f40089885747aa6c4c635 P 968c06ee65964aceb131114dfbd683de: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.620604 439 tablet_bootstrap.cc:492] T 91a40253e73f40089885747aa6c4c635 P 968c06ee65964aceb131114dfbd683de: No bootstrap required, opened a new log
I20240502 01:52:33.620734 439 ts_tablet_manager.cc:1367] T 91a40253e73f40089885747aa6c4c635 P 968c06ee65964aceb131114dfbd683de: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:52:33.621450 439 raft_consensus.cc:348] T 91a40253e73f40089885747aa6c4c635 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.621631 439 raft_consensus.cc:374] T 91a40253e73f40089885747aa6c4c635 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.621724 439 raft_consensus.cc:729] T 91a40253e73f40089885747aa6c4c635 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.621917 439 consensus_queue.cc:260] T 91a40253e73f40089885747aa6c4c635 P 968c06ee65964aceb131114dfbd683de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.622566 439 ts_tablet_manager.cc:1398] T 91a40253e73f40089885747aa6c4c635 P 968c06ee65964aceb131114dfbd683de: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:33.622828 439 tablet_bootstrap.cc:492] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de: Bootstrap starting.
I20240502 01:52:33.624388 439 tablet_bootstrap.cc:654] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.626375 437 tablet_bootstrap.cc:492] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5: No bootstrap required, opened a new log
I20240502 01:52:33.626545 437 ts_tablet_manager.cc:1367] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240502 01:52:33.626701 439 tablet_bootstrap.cc:492] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de: No bootstrap required, opened a new log
I20240502 01:52:33.626847 439 ts_tablet_manager.cc:1367] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:52:33.627301 437 raft_consensus.cc:348] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.627506 437 raft_consensus.cc:374] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.627583 437 raft_consensus.cc:729] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fc42d44d3441eaa2616635544066a5, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.627583 439 raft_consensus.cc:348] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.627777 439 raft_consensus.cc:374] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.627849 439 raft_consensus.cc:729] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.627852 437 consensus_queue.cc:260] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.628085 439 consensus_queue.cc:260] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.628379 438 tablet_bootstrap.cc:492] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325: No bootstrap required, opened a new log
I20240502 01:52:33.628532 438 ts_tablet_manager.cc:1367] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.012s
I20240502 01:52:33.628635 439 ts_tablet_manager.cc:1398] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:52:33.628757 437 ts_tablet_manager.cc:1398] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:52:33.628909 439 tablet_bootstrap.cc:492] T 29f468ee49544781b84c22fb48f57ede P 968c06ee65964aceb131114dfbd683de: Bootstrap starting.
I20240502 01:52:33.629042 437 tablet_bootstrap.cc:492] T 9e11386999d543a99ca5b08059e3048f P 50fc42d44d3441eaa2616635544066a5: Bootstrap starting.
I20240502 01:52:33.629272 438 raft_consensus.cc:348] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.629515 438 raft_consensus.cc:374] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.629606 438 raft_consensus.cc:729] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b0c49d96854b1a8f43e149053ca325, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.629842 438 consensus_queue.cc:260] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.630400 439 tablet_bootstrap.cc:654] T 29f468ee49544781b84c22fb48f57ede P 968c06ee65964aceb131114dfbd683de: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.630465 438 ts_tablet_manager.cc:1398] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:33.630498 437 tablet_bootstrap.cc:654] T 9e11386999d543a99ca5b08059e3048f P 50fc42d44d3441eaa2616635544066a5: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.630729 438 tablet_bootstrap.cc:492] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325: Bootstrap starting.
I20240502 01:52:33.632407 438 tablet_bootstrap.cc:654] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.633718 439 tablet_bootstrap.cc:492] T 29f468ee49544781b84c22fb48f57ede P 968c06ee65964aceb131114dfbd683de: No bootstrap required, opened a new log
I20240502 01:52:33.633852 439 ts_tablet_manager.cc:1367] T 29f468ee49544781b84c22fb48f57ede P 968c06ee65964aceb131114dfbd683de: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:52:33.634562 439 raft_consensus.cc:348] T 29f468ee49544781b84c22fb48f57ede P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.634737 439 raft_consensus.cc:374] T 29f468ee49544781b84c22fb48f57ede P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.634824 439 raft_consensus.cc:729] T 29f468ee49544781b84c22fb48f57ede P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.635030 439 consensus_queue.cc:260] T 29f468ee49544781b84c22fb48f57ede P 968c06ee65964aceb131114dfbd683de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.635578 439 ts_tablet_manager.cc:1398] T 29f468ee49544781b84c22fb48f57ede P 968c06ee65964aceb131114dfbd683de: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:33.635821 439 tablet_bootstrap.cc:492] T 626b95f7b22d4c278afda157cf50f7cc P 968c06ee65964aceb131114dfbd683de: Bootstrap starting.
I20240502 01:52:33.637267 439 tablet_bootstrap.cc:654] T 626b95f7b22d4c278afda157cf50f7cc P 968c06ee65964aceb131114dfbd683de: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.639483 437 tablet_bootstrap.cc:492] T 9e11386999d543a99ca5b08059e3048f P 50fc42d44d3441eaa2616635544066a5: No bootstrap required, opened a new log
I20240502 01:52:33.639612 437 ts_tablet_manager.cc:1367] T 9e11386999d543a99ca5b08059e3048f P 50fc42d44d3441eaa2616635544066a5: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20240502 01:52:33.639681 439 tablet_bootstrap.cc:492] T 626b95f7b22d4c278afda157cf50f7cc P 968c06ee65964aceb131114dfbd683de: No bootstrap required, opened a new log
I20240502 01:52:33.639842 439 ts_tablet_manager.cc:1367] T 626b95f7b22d4c278afda157cf50f7cc P 968c06ee65964aceb131114dfbd683de: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:52:33.640197 438 tablet_bootstrap.cc:492] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325: No bootstrap required, opened a new log
I20240502 01:52:33.640229 437 raft_consensus.cc:348] T 9e11386999d543a99ca5b08059e3048f P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.640367 438 ts_tablet_manager.cc:1367] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240502 01:52:33.640401 437 raft_consensus.cc:374] T 9e11386999d543a99ca5b08059e3048f P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.640491 437 raft_consensus.cc:729] T 9e11386999d543a99ca5b08059e3048f P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fc42d44d3441eaa2616635544066a5, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.640506 439 raft_consensus.cc:348] T 626b95f7b22d4c278afda157cf50f7cc P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.640712 439 raft_consensus.cc:374] T 626b95f7b22d4c278afda157cf50f7cc P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.640787 439 raft_consensus.cc:729] T 626b95f7b22d4c278afda157cf50f7cc P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.640726 437 consensus_queue.cc:260] T 9e11386999d543a99ca5b08059e3048f P 50fc42d44d3441eaa2616635544066a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.641034 439 consensus_queue.cc:260] T 626b95f7b22d4c278afda157cf50f7cc P 968c06ee65964aceb131114dfbd683de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.641119 438 raft_consensus.cc:348] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.641333 437 ts_tablet_manager.cc:1398] T 9e11386999d543a99ca5b08059e3048f P 50fc42d44d3441eaa2616635544066a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:52:33.641419 438 raft_consensus.cc:374] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.641502 438 raft_consensus.cc:729] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b0c49d96854b1a8f43e149053ca325, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.641588 437 tablet_bootstrap.cc:492] T fd7160232773423590368d00a1cb8e88 P 50fc42d44d3441eaa2616635544066a5: Bootstrap starting.
I20240502 01:52:33.641808 438 consensus_queue.cc:260] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.642383 438 ts_tablet_manager.cc:1398] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:33.642654 438 tablet_bootstrap.cc:492] T 10898226a6644583915e21ce167afbed P 79b0c49d96854b1a8f43e149053ca325: Bootstrap starting.
I20240502 01:52:33.643095 437 tablet_bootstrap.cc:654] T fd7160232773423590368d00a1cb8e88 P 50fc42d44d3441eaa2616635544066a5: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.643110 439 ts_tablet_manager.cc:1398] T 626b95f7b22d4c278afda157cf50f7cc P 968c06ee65964aceb131114dfbd683de: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 01:52:33.643421 439 tablet_bootstrap.cc:492] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de: Bootstrap starting.
I20240502 01:52:33.644210 438 tablet_bootstrap.cc:654] T 10898226a6644583915e21ce167afbed P 79b0c49d96854b1a8f43e149053ca325: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.644877 439 tablet_bootstrap.cc:654] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de: Neither blocks nor log segments found. Creating new log.
W20240502 01:52:33.650561 32756 tablet.cc:2369] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:52:33.650996 437 tablet_bootstrap.cc:492] T fd7160232773423590368d00a1cb8e88 P 50fc42d44d3441eaa2616635544066a5: No bootstrap required, opened a new log
I20240502 01:52:33.651168 437 ts_tablet_manager.cc:1367] T fd7160232773423590368d00a1cb8e88 P 50fc42d44d3441eaa2616635544066a5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240502 01:52:33.651945 437 raft_consensus.cc:348] T fd7160232773423590368d00a1cb8e88 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.652153 437 raft_consensus.cc:374] T fd7160232773423590368d00a1cb8e88 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.652225 437 raft_consensus.cc:729] T fd7160232773423590368d00a1cb8e88 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fc42d44d3441eaa2616635544066a5, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.652474 437 consensus_queue.cc:260] T fd7160232773423590368d00a1cb8e88 P 50fc42d44d3441eaa2616635544066a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.653101 437 ts_tablet_manager.cc:1398] T fd7160232773423590368d00a1cb8e88 P 50fc42d44d3441eaa2616635544066a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:33.653393 437 tablet_bootstrap.cc:492] T 06e288c44ec5454caade0f9fd5b86822 P 50fc42d44d3441eaa2616635544066a5: Bootstrap starting.
I20240502 01:52:33.655083 437 tablet_bootstrap.cc:654] T 06e288c44ec5454caade0f9fd5b86822 P 50fc42d44d3441eaa2616635544066a5: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.656988 438 tablet_bootstrap.cc:492] T 10898226a6644583915e21ce167afbed P 79b0c49d96854b1a8f43e149053ca325: No bootstrap required, opened a new log
I20240502 01:52:33.657157 438 ts_tablet_manager.cc:1367] T 10898226a6644583915e21ce167afbed P 79b0c49d96854b1a8f43e149053ca325: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.008s
I20240502 01:52:33.657606 439 tablet_bootstrap.cc:492] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de: No bootstrap required, opened a new log
I20240502 01:52:33.657757 439 ts_tablet_manager.cc:1367] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.009s
I20240502 01:52:33.657902 437 tablet_bootstrap.cc:492] T 06e288c44ec5454caade0f9fd5b86822 P 50fc42d44d3441eaa2616635544066a5: No bootstrap required, opened a new log
I20240502 01:52:33.658026 437 ts_tablet_manager.cc:1367] T 06e288c44ec5454caade0f9fd5b86822 P 50fc42d44d3441eaa2616635544066a5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:52:33.657914 438 raft_consensus.cc:348] T 10898226a6644583915e21ce167afbed P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.658097 438 raft_consensus.cc:374] T 10898226a6644583915e21ce167afbed P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.658186 438 raft_consensus.cc:729] T 10898226a6644583915e21ce167afbed P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b0c49d96854b1a8f43e149053ca325, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.658448 438 consensus_queue.cc:260] T 10898226a6644583915e21ce167afbed P 79b0c49d96854b1a8f43e149053ca325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.658504 439 raft_consensus.cc:348] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.658672 439 raft_consensus.cc:374] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.658771 439 raft_consensus.cc:729] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.659061 438 ts_tablet_manager.cc:1398] T 10898226a6644583915e21ce167afbed P 79b0c49d96854b1a8f43e149053ca325: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:33.658630 437 raft_consensus.cc:348] T 06e288c44ec5454caade0f9fd5b86822 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.659046 439 consensus_queue.cc:260] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.659225 437 raft_consensus.cc:374] T 06e288c44ec5454caade0f9fd5b86822 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.659313 438 tablet_bootstrap.cc:492] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325: Bootstrap starting.
I20240502 01:52:33.659651 439 ts_tablet_manager.cc:1398] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:33.659932 439 tablet_bootstrap.cc:492] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de: Bootstrap starting.
I20240502 01:52:33.660825 437 raft_consensus.cc:729] T 06e288c44ec5454caade0f9fd5b86822 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fc42d44d3441eaa2616635544066a5, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.661051 437 consensus_queue.cc:260] T 06e288c44ec5454caade0f9fd5b86822 P 50fc42d44d3441eaa2616635544066a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.661417 439 tablet_bootstrap.cc:654] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.661590 437 ts_tablet_manager.cc:1398] T 06e288c44ec5454caade0f9fd5b86822 P 50fc42d44d3441eaa2616635544066a5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:52:33.661880 437 tablet_bootstrap.cc:492] T 29f468ee49544781b84c22fb48f57ede P 50fc42d44d3441eaa2616635544066a5: Bootstrap starting.
I20240502 01:52:33.663422 437 tablet_bootstrap.cc:654] T 29f468ee49544781b84c22fb48f57ede P 50fc42d44d3441eaa2616635544066a5: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.664359 438 tablet_bootstrap.cc:654] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.666373 438 tablet_bootstrap.cc:492] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325: No bootstrap required, opened a new log
I20240502 01:52:33.666513 438 ts_tablet_manager.cc:1367] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 01:52:33.667105 438 raft_consensus.cc:348] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.667277 438 raft_consensus.cc:374] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.667361 438 raft_consensus.cc:729] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b0c49d96854b1a8f43e149053ca325, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.667611 438 consensus_queue.cc:260] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.668094 438 ts_tablet_manager.cc:1398] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:52:33.668404 438 tablet_bootstrap.cc:492] T 657f6ad36cf34cbcaca11cd331672993 P 79b0c49d96854b1a8f43e149053ca325: Bootstrap starting.
I20240502 01:52:33.668519 437 tablet_bootstrap.cc:492] T 29f468ee49544781b84c22fb48f57ede P 50fc42d44d3441eaa2616635544066a5: No bootstrap required, opened a new log
I20240502 01:52:33.668661 437 ts_tablet_manager.cc:1367] T 29f468ee49544781b84c22fb48f57ede P 50fc42d44d3441eaa2616635544066a5: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240502 01:52:33.669210 437 raft_consensus.cc:348] T 29f468ee49544781b84c22fb48f57ede P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.669440 437 raft_consensus.cc:374] T 29f468ee49544781b84c22fb48f57ede P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.669521 437 raft_consensus.cc:729] T 29f468ee49544781b84c22fb48f57ede P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fc42d44d3441eaa2616635544066a5, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.669754 437 consensus_queue.cc:260] T 29f468ee49544781b84c22fb48f57ede P 50fc42d44d3441eaa2616635544066a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.670127 439 tablet_bootstrap.cc:492] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de: No bootstrap required, opened a new log
I20240502 01:52:33.670317 439 ts_tablet_manager.cc:1367] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240502 01:52:33.670333 437 ts_tablet_manager.cc:1398] T 29f468ee49544781b84c22fb48f57ede P 50fc42d44d3441eaa2616635544066a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:33.670464 438 tablet_bootstrap.cc:654] T 657f6ad36cf34cbcaca11cd331672993 P 79b0c49d96854b1a8f43e149053ca325: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.670626 437 tablet_bootstrap.cc:492] T a153b9fa8359422eb28faa8f573edb58 P 50fc42d44d3441eaa2616635544066a5: Bootstrap starting.
I20240502 01:52:33.671133 439 raft_consensus.cc:348] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.671428 439 raft_consensus.cc:374] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.671540 439 raft_consensus.cc:729] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.671839 439 consensus_queue.cc:260] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.672049 437 tablet_bootstrap.cc:654] T a153b9fa8359422eb28faa8f573edb58 P 50fc42d44d3441eaa2616635544066a5: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.672497 439 ts_tablet_manager.cc:1398] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:52:33.672859 439 tablet_bootstrap.cc:492] T 79527d85ba284c04ae435ed71a3d2d0c P 968c06ee65964aceb131114dfbd683de: Bootstrap starting.
I20240502 01:52:33.674461 439 tablet_bootstrap.cc:654] T 79527d85ba284c04ae435ed71a3d2d0c P 968c06ee65964aceb131114dfbd683de: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.676950 437 tablet_bootstrap.cc:492] T a153b9fa8359422eb28faa8f573edb58 P 50fc42d44d3441eaa2616635544066a5: No bootstrap required, opened a new log
I20240502 01:52:33.677109 437 ts_tablet_manager.cc:1367] T a153b9fa8359422eb28faa8f573edb58 P 50fc42d44d3441eaa2616635544066a5: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20240502 01:52:33.677819 437 raft_consensus.cc:348] T a153b9fa8359422eb28faa8f573edb58 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.678016 437 raft_consensus.cc:374] T a153b9fa8359422eb28faa8f573edb58 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.678104 437 raft_consensus.cc:729] T a153b9fa8359422eb28faa8f573edb58 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fc42d44d3441eaa2616635544066a5, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.678316 437 consensus_queue.cc:260] T a153b9fa8359422eb28faa8f573edb58 P 50fc42d44d3441eaa2616635544066a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.678889 437 ts_tablet_manager.cc:1398] T a153b9fa8359422eb28faa8f573edb58 P 50fc42d44d3441eaa2616635544066a5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:52:33.679178 437 tablet_bootstrap.cc:492] T cf049ada294349e3b3811e1397d4851d P 50fc42d44d3441eaa2616635544066a5: Bootstrap starting.
I20240502 01:52:33.679469 443 raft_consensus.cc:482] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:33.679708 443 raft_consensus.cc:504] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } }
I20240502 01:52:33.680872 437 tablet_bootstrap.cc:654] T cf049ada294349e3b3811e1397d4851d P 50fc42d44d3441eaa2616635544066a5: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.681982 443 leader_election.cc:288] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451), 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847)
I20240502 01:52:33.683013 438 tablet_bootstrap.cc:492] T 657f6ad36cf34cbcaca11cd331672993 P 79b0c49d96854b1a8f43e149053ca325: No bootstrap required, opened a new log
I20240502 01:52:33.683169 438 ts_tablet_manager.cc:1367] T 657f6ad36cf34cbcaca11cd331672993 P 79b0c49d96854b1a8f43e149053ca325: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.012s
I20240502 01:52:33.683234 444 raft_consensus.cc:482] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:33.683504 444 raft_consensus.cc:504] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.684032 438 raft_consensus.cc:348] T 657f6ad36cf34cbcaca11cd331672993 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } }
I20240502 01:52:33.684240 438 raft_consensus.cc:374] T 657f6ad36cf34cbcaca11cd331672993 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.684355 438 raft_consensus.cc:729] T 657f6ad36cf34cbcaca11cd331672993 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b0c49d96854b1a8f43e149053ca325, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.684931 438 consensus_queue.cc:260] T 657f6ad36cf34cbcaca11cd331672993 P 79b0c49d96854b1a8f43e149053ca325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } }
I20240502 01:52:33.685539 444 leader_election.cc:288] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619), 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451)
I20240502 01:52:33.686103 437 tablet_bootstrap.cc:492] T cf049ada294349e3b3811e1397d4851d P 50fc42d44d3441eaa2616635544066a5: No bootstrap required, opened a new log
I20240502 01:52:33.686250 437 ts_tablet_manager.cc:1367] T cf049ada294349e3b3811e1397d4851d P 50fc42d44d3441eaa2616635544066a5: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240502 01:52:33.687016 437 raft_consensus.cc:348] T cf049ada294349e3b3811e1397d4851d P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.687230 437 raft_consensus.cc:374] T cf049ada294349e3b3811e1397d4851d P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.687291 437 raft_consensus.cc:729] T cf049ada294349e3b3811e1397d4851d P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fc42d44d3441eaa2616635544066a5, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.687511 437 consensus_queue.cc:260] T cf049ada294349e3b3811e1397d4851d P 50fc42d44d3441eaa2616635544066a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.687973 437 ts_tablet_manager.cc:1398] T cf049ada294349e3b3811e1397d4851d P 50fc42d44d3441eaa2616635544066a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:52:33.688241 437 tablet_bootstrap.cc:492] T 91a40253e73f40089885747aa6c4c635 P 50fc42d44d3441eaa2616635544066a5: Bootstrap starting.
I20240502 01:52:33.688235 444 raft_consensus.cc:482] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:33.688396 444 raft_consensus.cc:504] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.688946 444 leader_election.cc:288] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619), 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451)
I20240502 01:52:33.689853 437 tablet_bootstrap.cc:654] T 91a40253e73f40089885747aa6c4c635 P 50fc42d44d3441eaa2616635544066a5: Neither blocks nor log segments found. Creating new log.
W20240502 01:52:33.690021 32625 tablet.cc:2369] T 10898226a6644583915e21ce167afbed P 79b0c49d96854b1a8f43e149053ca325: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:52:33.691735 439 tablet_bootstrap.cc:492] T 79527d85ba284c04ae435ed71a3d2d0c P 968c06ee65964aceb131114dfbd683de: No bootstrap required, opened a new log
I20240502 01:52:33.691895 439 ts_tablet_manager.cc:1367] T 79527d85ba284c04ae435ed71a3d2d0c P 968c06ee65964aceb131114dfbd683de: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.007s
I20240502 01:52:33.693133 439 raft_consensus.cc:348] T 79527d85ba284c04ae435ed71a3d2d0c P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.693333 439 raft_consensus.cc:374] T 79527d85ba284c04ae435ed71a3d2d0c P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.693418 439 raft_consensus.cc:729] T 79527d85ba284c04ae435ed71a3d2d0c P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.693663 439 consensus_queue.cc:260] T 79527d85ba284c04ae435ed71a3d2d0c P 968c06ee65964aceb131114dfbd683de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.694341 439 ts_tablet_manager.cc:1398] T 79527d85ba284c04ae435ed71a3d2d0c P 968c06ee65964aceb131114dfbd683de: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:52:33.694630 439 tablet_bootstrap.cc:492] T b9f486c135914f558f3c847ac02dda8a P 968c06ee65964aceb131114dfbd683de: Bootstrap starting.
I20240502 01:52:33.696220 439 tablet_bootstrap.cc:654] T b9f486c135914f558f3c847ac02dda8a P 968c06ee65964aceb131114dfbd683de: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.706336 447 raft_consensus.cc:482] T 91a40253e73f40089885747aa6c4c635 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:33.706588 447 raft_consensus.cc:504] T 91a40253e73f40089885747aa6c4c635 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.708609 447 leader_election.cc:288] T 91a40253e73f40089885747aa6c4c635 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619), 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847)
I20240502 01:52:33.708840 447 raft_consensus.cc:482] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:33.708985 447 raft_consensus.cc:504] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.709542 447 leader_election.cc:288] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619), 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847)
I20240502 01:52:33.724943 443 raft_consensus.cc:482] T cf049ada294349e3b3811e1397d4851d P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:33.725169 443 raft_consensus.cc:504] T cf049ada294349e3b3811e1397d4851d P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.725829 443 leader_election.cc:288] T cf049ada294349e3b3811e1397d4851d P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847), 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451)
I20240502 01:52:33.734396 438 ts_tablet_manager.cc:1398] T 657f6ad36cf34cbcaca11cd331672993 P 79b0c49d96854b1a8f43e149053ca325: Time spent starting tablet: real 0.051s user 0.012s sys 0.012s
I20240502 01:52:33.734789 438 tablet_bootstrap.cc:492] T b9f486c135914f558f3c847ac02dda8a P 79b0c49d96854b1a8f43e149053ca325: Bootstrap starting.
I20240502 01:52:33.736811 438 tablet_bootstrap.cc:654] T b9f486c135914f558f3c847ac02dda8a P 79b0c49d96854b1a8f43e149053ca325: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.740718 437 tablet_bootstrap.cc:492] T 91a40253e73f40089885747aa6c4c635 P 50fc42d44d3441eaa2616635544066a5: No bootstrap required, opened a new log
I20240502 01:52:33.740942 437 ts_tablet_manager.cc:1367] T 91a40253e73f40089885747aa6c4c635 P 50fc42d44d3441eaa2616635544066a5: Time spent bootstrapping tablet: real 0.053s user 0.008s sys 0.017s
I20240502 01:52:33.742373 437 raft_consensus.cc:348] T 91a40253e73f40089885747aa6c4c635 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.742669 437 raft_consensus.cc:374] T 91a40253e73f40089885747aa6c4c635 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.742794 437 raft_consensus.cc:729] T 91a40253e73f40089885747aa6c4c635 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fc42d44d3441eaa2616635544066a5, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.742724 32709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f468ee49544781b84c22fb48f57ede" candidate_uuid: "79b0c49d96854b1a8f43e149053ca325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5" is_pre_election: true
I20240502 01:52:33.743229 32709 raft_consensus.cc:2454] T 29f468ee49544781b84c22fb48f57ede P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79b0c49d96854b1a8f43e149053ca325 in term 0.
I20240502 01:52:33.743113 437 consensus_queue.cc:260] T 91a40253e73f40089885747aa6c4c635 P 50fc42d44d3441eaa2616635544066a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.744210 32709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626b95f7b22d4c278afda157cf50f7cc" candidate_uuid: "79b0c49d96854b1a8f43e149053ca325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5" is_pre_election: true
I20240502 01:52:33.744428 437 ts_tablet_manager.cc:1398] T 91a40253e73f40089885747aa6c4c635 P 50fc42d44d3441eaa2616635544066a5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:52:33.744726 437 tablet_bootstrap.cc:492] T 10898226a6644583915e21ce167afbed P 50fc42d44d3441eaa2616635544066a5: Bootstrap starting.
I20240502 01:52:33.746428 479 raft_consensus.cc:482] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:33.746624 479 raft_consensus.cc:504] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.747561 479 leader_election.cc:288] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847), 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619)
I20240502 01:52:33.748621 437 tablet_bootstrap.cc:654] T 10898226a6644583915e21ce167afbed P 50fc42d44d3441eaa2616635544066a5: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.741838 473 raft_consensus.cc:482] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:33.745354 32511 leader_election.cc:302] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 79b0c49d96854b1a8f43e149053ca325; no voters:
I20240502 01:52:33.745761 438 tablet_bootstrap.cc:492] T b9f486c135914f558f3c847ac02dda8a P 79b0c49d96854b1a8f43e149053ca325: No bootstrap required, opened a new log
I20240502 01:52:33.749204 438 ts_tablet_manager.cc:1367] T b9f486c135914f558f3c847ac02dda8a P 79b0c49d96854b1a8f43e149053ca325: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.001s
I20240502 01:52:33.749473 444 raft_consensus.cc:2781] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:33.749641 444 raft_consensus.cc:482] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240502 01:52:33.749662 32511 leader_election.cc:341] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:33.749840 444 raft_consensus.cc:3037] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.749946 473 raft_consensus.cc:504] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.750075 438 raft_consensus.cc:348] T b9f486c135914f558f3c847ac02dda8a P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.750360 438 raft_consensus.cc:374] T b9f486c135914f558f3c847ac02dda8a P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.750437 438 raft_consensus.cc:729] T b9f486c135914f558f3c847ac02dda8a P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b0c49d96854b1a8f43e149053ca325, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.751106 32709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a40253e73f40089885747aa6c4c635" candidate_uuid: "79b0c49d96854b1a8f43e149053ca325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5" is_pre_election: true
I20240502 01:52:33.751336 32709 raft_consensus.cc:2454] T 91a40253e73f40089885747aa6c4c635 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79b0c49d96854b1a8f43e149053ca325 in term 0.
I20240502 01:52:33.751765 32511 leader_election.cc:302] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 79b0c49d96854b1a8f43e149053ca325; no voters:
I20240502 01:52:33.752110 473 leader_election.cc:288] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619), 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451)
I20240502 01:52:33.752292 473 raft_consensus.cc:2781] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:33.752409 473 raft_consensus.cc:482] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:33.752488 473 raft_consensus.cc:3037] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.750711 438 consensus_queue.cc:260] T b9f486c135914f558f3c847ac02dda8a P 79b0c49d96854b1a8f43e149053ca325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.752885 444 raft_consensus.cc:504] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.753234 438 ts_tablet_manager.cc:1398] T b9f486c135914f558f3c847ac02dda8a P 79b0c49d96854b1a8f43e149053ca325: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:52:33.753386 479 raft_consensus.cc:482] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:33.753572 479 raft_consensus.cc:504] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.754168 444 leader_election.cc:288] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325 [CANDIDATE]: Term 1 election: Requested vote from peers 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619), 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451)
I20240502 01:52:33.754182 479 leader_election.cc:288] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847), 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619)
I20240502 01:52:33.754555 438 tablet_bootstrap.cc:492] T 9e11386999d543a99ca5b08059e3048f P 79b0c49d96854b1a8f43e149053ca325: Bootstrap starting.
I20240502 01:52:33.754447 372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f468ee49544781b84c22fb48f57ede" candidate_uuid: "79b0c49d96854b1a8f43e149053ca325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c06ee65964aceb131114dfbd683de" is_pre_election: true
I20240502 01:52:33.754880 372 raft_consensus.cc:2454] T 29f468ee49544781b84c22fb48f57ede P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79b0c49d96854b1a8f43e149053ca325 in term 0.
I20240502 01:52:33.755637 372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626b95f7b22d4c278afda157cf50f7cc" candidate_uuid: "79b0c49d96854b1a8f43e149053ca325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c06ee65964aceb131114dfbd683de" is_pre_election: true
I20240502 01:52:33.755815 372 raft_consensus.cc:2454] T 626b95f7b22d4c278afda157cf50f7cc P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79b0c49d96854b1a8f43e149053ca325 in term 0.
I20240502 01:52:33.756345 438 tablet_bootstrap.cc:654] T 9e11386999d543a99ca5b08059e3048f P 79b0c49d96854b1a8f43e149053ca325: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.756311 372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a40253e73f40089885747aa6c4c635" candidate_uuid: "79b0c49d96854b1a8f43e149053ca325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c06ee65964aceb131114dfbd683de" is_pre_election: true
I20240502 01:52:33.756481 372 raft_consensus.cc:2454] T 91a40253e73f40089885747aa6c4c635 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79b0c49d96854b1a8f43e149053ca325 in term 0.
I20240502 01:52:33.756947 372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f468ee49544781b84c22fb48f57ede" candidate_uuid: "79b0c49d96854b1a8f43e149053ca325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c06ee65964aceb131114dfbd683de"
I20240502 01:52:33.757122 372 raft_consensus.cc:3037] T 29f468ee49544781b84c22fb48f57ede P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.760425 372 raft_consensus.cc:2454] T 29f468ee49544781b84c22fb48f57ede P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79b0c49d96854b1a8f43e149053ca325 in term 1.
I20240502 01:52:33.760946 32709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29f468ee49544781b84c22fb48f57ede" candidate_uuid: "79b0c49d96854b1a8f43e149053ca325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5"
I20240502 01:52:33.761183 32709 raft_consensus.cc:3037] T 29f468ee49544781b84c22fb48f57ede P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.762210 473 raft_consensus.cc:504] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.764367 32709 raft_consensus.cc:2454] T 29f468ee49544781b84c22fb48f57ede P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79b0c49d96854b1a8f43e149053ca325 in term 1.
I20240502 01:52:33.765146 32709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a40253e73f40089885747aa6c4c635" candidate_uuid: "79b0c49d96854b1a8f43e149053ca325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5"
I20240502 01:52:33.765394 32709 raft_consensus.cc:3037] T 91a40253e73f40089885747aa6c4c635 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.768378 32709 raft_consensus.cc:2454] T 91a40253e73f40089885747aa6c4c635 P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79b0c49d96854b1a8f43e149053ca325 in term 1.
I20240502 01:52:33.768646 32511 leader_election.cc:302] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 79b0c49d96854b1a8f43e149053ca325; no voters:
I20240502 01:52:33.768975 485 raft_consensus.cc:2781] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:33.769008 444 raft_consensus.cc:482] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:33.769157 444 raft_consensus.cc:504] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.769781 444 leader_election.cc:288] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619), 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451)
I20240502 01:52:33.771385 372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf049ada294349e3b3811e1397d4851d" candidate_uuid: "79b0c49d96854b1a8f43e149053ca325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c06ee65964aceb131114dfbd683de" is_pre_election: true
I20240502 01:52:33.772552 32512 leader_election.cc:302] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 79b0c49d96854b1a8f43e149053ca325, 968c06ee65964aceb131114dfbd683de; no voters: 50fc42d44d3441eaa2616635544066a5
I20240502 01:52:33.773469 444 raft_consensus.cc:2781] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:33.773633 444 raft_consensus.cc:482] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:33.773795 444 raft_consensus.cc:3037] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.776682 444 raft_consensus.cc:504] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.770375 32708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf049ada294349e3b3811e1397d4851d" candidate_uuid: "79b0c49d96854b1a8f43e149053ca325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5" is_pre_election: true
I20240502 01:52:33.777828 32708 raft_consensus.cc:2454] T cf049ada294349e3b3811e1397d4851d P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79b0c49d96854b1a8f43e149053ca325 in term 0.
I20240502 01:52:33.777701 32707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626b95f7b22d4c278afda157cf50f7cc" candidate_uuid: "79b0c49d96854b1a8f43e149053ca325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5"
I20240502 01:52:33.778556 32511 leader_election.cc:302] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 79b0c49d96854b1a8f43e149053ca325; no voters:
W20240502 01:52:33.778890 32511 leader_election.cc:341] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:33.779119 32511 leader_election.cc:302] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 79b0c49d96854b1a8f43e149053ca325; no voters:
W20240502 01:52:33.779515 32512 leader_election.cc:341] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:33.780269 372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a40253e73f40089885747aa6c4c635" candidate_uuid: "79b0c49d96854b1a8f43e149053ca325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c06ee65964aceb131114dfbd683de"
I20240502 01:52:33.780468 372 raft_consensus.cc:3037] T 91a40253e73f40089885747aa6c4c635 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.783277 372 raft_consensus.cc:2454] T 91a40253e73f40089885747aa6c4c635 P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79b0c49d96854b1a8f43e149053ca325 in term 1.
I20240502 01:52:33.783455 32578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a40253e73f40089885747aa6c4c635" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325" is_pre_election: true
I20240502 01:52:33.783946 32578 raft_consensus.cc:2379] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 968c06ee65964aceb131114dfbd683de in current term 1: Already voted for candidate 79b0c49d96854b1a8f43e149053ca325 in this term.
I20240502 01:52:33.784263 473 leader_election.cc:288] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325 [CANDIDATE]: Term 1 election: Requested vote from peers 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619), 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451)
I20240502 01:52:33.784466 473 raft_consensus.cc:2781] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:33.784546 473 raft_consensus.cc:482] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:33.784662 473 raft_consensus.cc:3037] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.784708 485 raft_consensus.cc:686] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325 [term 1 LEADER]: Becoming Leader. State: Replica: 79b0c49d96854b1a8f43e149053ca325, State: Running, Role: LEADER
I20240502 01:52:33.785099 32578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a153b9fa8359422eb28faa8f573edb58" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325" is_pre_election: true
I20240502 01:52:33.785162 479 raft_consensus.cc:482] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:33.785283 479 raft_consensus.cc:504] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.785315 485 consensus_queue.cc:237] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.785879 479 leader_election.cc:288] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619), 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847)
I20240502 01:52:33.786152 32578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e11386999d543a99ca5b08059e3048f" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325" is_pre_election: true
I20240502 01:52:33.786909 32578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd7160232773423590368d00a1cb8e88" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325" is_pre_election: true
I20240502 01:52:33.787093 32578 raft_consensus.cc:2454] T fd7160232773423590368d00a1cb8e88 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 0.
I20240502 01:52:33.787889 437 tablet_bootstrap.cc:492] T 10898226a6644583915e21ce167afbed P 50fc42d44d3441eaa2616635544066a5: No bootstrap required, opened a new log
I20240502 01:52:33.787942 32578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10898226a6644583915e21ce167afbed" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325" is_pre_election: true
I20240502 01:52:33.788064 437 ts_tablet_manager.cc:1367] T 10898226a6644583915e21ce167afbed P 50fc42d44d3441eaa2616635544066a5: Time spent bootstrapping tablet: real 0.043s user 0.011s sys 0.005s
I20240502 01:52:33.788105 32578 raft_consensus.cc:2454] T 10898226a6644583915e21ce167afbed P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 0.
I20240502 01:52:33.788350 488 raft_consensus.cc:2781] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:33.788489 488 raft_consensus.cc:686] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325 [term 1 LEADER]: Becoming Leader. State: Replica: 79b0c49d96854b1a8f43e149053ca325, State: Running, Role: LEADER
I20240502 01:52:33.788852 437 raft_consensus.cc:348] T 10898226a6644583915e21ce167afbed P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.789103 437 raft_consensus.cc:374] T 10898226a6644583915e21ce167afbed P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.789201 437 raft_consensus.cc:729] T 10898226a6644583915e21ce167afbed P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fc42d44d3441eaa2616635544066a5, State: Initialized, Role: FOLLOWER
W20240502 01:52:33.788988 306 leader_election.cc:341] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:33.789450 437 consensus_queue.cc:260] T 10898226a6644583915e21ce167afbed P 50fc42d44d3441eaa2616635544066a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
W20240502 01:52:33.789696 306 leader_election.cc:341] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:33.789927 306 leader_election.cc:302] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79b0c49d96854b1a8f43e149053ca325, 968c06ee65964aceb131114dfbd683de; no voters:
I20240502 01:52:33.790342 479 raft_consensus.cc:2781] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:33.788786 488 consensus_queue.cc:237] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.790448 479 raft_consensus.cc:482] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:33.790526 479 raft_consensus.cc:3037] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.790540 437 ts_tablet_manager.cc:1398] T 10898226a6644583915e21ce167afbed P 50fc42d44d3441eaa2616635544066a5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:52:33.790840 437 tablet_bootstrap.cc:492] T 626b95f7b22d4c278afda157cf50f7cc P 50fc42d44d3441eaa2616635544066a5: Bootstrap starting.
I20240502 01:52:33.793493 479 raft_consensus.cc:504] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.794279 479 leader_election.cc:288] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 election: Requested vote from peers 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847), 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619)
I20240502 01:52:33.794577 437 tablet_bootstrap.cc:654] T 626b95f7b22d4c278afda157cf50f7cc P 50fc42d44d3441eaa2616635544066a5: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.794613 306 leader_election.cc:302] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79b0c49d96854b1a8f43e149053ca325, 968c06ee65964aceb131114dfbd683de; no voters:
I20240502 01:52:33.795105 473 raft_consensus.cc:504] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.795523 479 raft_consensus.cc:2781] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:33.795614 479 raft_consensus.cc:482] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:33.795696 479 raft_consensus.cc:3037] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.798229 479 raft_consensus.cc:504] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.798983 479 leader_election.cc:288] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 election: Requested vote from peers 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619), 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847)
I20240502 01:52:33.799636 32578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd7160232773423590368d00a1cb8e88" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325"
I20240502 01:52:33.799818 32578 raft_consensus.cc:3037] T fd7160232773423590368d00a1cb8e88 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.800477 473 leader_election.cc:288] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325 [CANDIDATE]: Term 1 election: Requested vote from peers 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619), 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451)
I20240502 01:52:33.802486 437 tablet_bootstrap.cc:492] T 626b95f7b22d4c278afda157cf50f7cc P 50fc42d44d3441eaa2616635544066a5: No bootstrap required, opened a new log
I20240502 01:52:33.802624 437 ts_tablet_manager.cc:1367] T 626b95f7b22d4c278afda157cf50f7cc P 50fc42d44d3441eaa2616635544066a5: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20240502 01:52:33.802757 32578 raft_consensus.cc:2454] T fd7160232773423590368d00a1cb8e88 P 79b0c49d96854b1a8f43e149053ca325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 1.
I20240502 01:52:33.803287 444 leader_election.cc:288] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325 [CANDIDATE]: Term 1 election: Requested vote from peers 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619), 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451)
I20240502 01:52:33.803359 437 raft_consensus.cc:348] T 626b95f7b22d4c278afda157cf50f7cc P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.803586 437 raft_consensus.cc:374] T 626b95f7b22d4c278afda157cf50f7cc P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.803678 437 raft_consensus.cc:729] T 626b95f7b22d4c278afda157cf50f7cc P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fc42d44d3441eaa2616635544066a5, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.803947 32578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10898226a6644583915e21ce167afbed" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325"
I20240502 01:52:33.804368 32578 raft_consensus.cc:3037] T 10898226a6644583915e21ce167afbed P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.805513 437 consensus_queue.cc:260] T 626b95f7b22d4c278afda157cf50f7cc P 50fc42d44d3441eaa2616635544066a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.807135 437 ts_tablet_manager.cc:1398] T 626b95f7b22d4c278afda157cf50f7cc P 50fc42d44d3441eaa2616635544066a5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:52:33.807425 437 tablet_bootstrap.cc:492] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 50fc42d44d3441eaa2616635544066a5: Bootstrap starting.
I20240502 01:52:33.807165 32578 raft_consensus.cc:2454] T 10898226a6644583915e21ce167afbed P 79b0c49d96854b1a8f43e149053ca325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 1.
I20240502 01:52:33.808744 372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626b95f7b22d4c278afda157cf50f7cc" candidate_uuid: "79b0c49d96854b1a8f43e149053ca325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c06ee65964aceb131114dfbd683de"
I20240502 01:52:33.808948 372 raft_consensus.cc:3037] T 626b95f7b22d4c278afda157cf50f7cc P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.809132 439 tablet_bootstrap.cc:492] T b9f486c135914f558f3c847ac02dda8a P 968c06ee65964aceb131114dfbd683de: No bootstrap required, opened a new log
I20240502 01:52:33.809286 439 ts_tablet_manager.cc:1367] T b9f486c135914f558f3c847ac02dda8a P 968c06ee65964aceb131114dfbd683de: Time spent bootstrapping tablet: real 0.115s user 0.021s sys 0.043s
I20240502 01:52:33.810055 439 raft_consensus.cc:348] T b9f486c135914f558f3c847ac02dda8a P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.810288 439 raft_consensus.cc:374] T b9f486c135914f558f3c847ac02dda8a P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.810355 439 raft_consensus.cc:729] T b9f486c135914f558f3c847ac02dda8a P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.810581 439 consensus_queue.cc:260] T b9f486c135914f558f3c847ac02dda8a P 968c06ee65964aceb131114dfbd683de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.811538 437 tablet_bootstrap.cc:654] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 50fc42d44d3441eaa2616635544066a5: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.811659 372 raft_consensus.cc:2454] T 626b95f7b22d4c278afda157cf50f7cc P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79b0c49d96854b1a8f43e149053ca325 in term 1.
I20240502 01:52:33.812358 32512 leader_election.cc:302] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 79b0c49d96854b1a8f43e149053ca325, 968c06ee65964aceb131114dfbd683de; no voters: 50fc42d44d3441eaa2616635544066a5
I20240502 01:52:33.812605 372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf049ada294349e3b3811e1397d4851d" candidate_uuid: "79b0c49d96854b1a8f43e149053ca325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c06ee65964aceb131114dfbd683de"
I20240502 01:52:33.812937 444 raft_consensus.cc:2781] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:33.813164 444 raft_consensus.cc:686] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325 [term 1 LEADER]: Becoming Leader. State: Replica: 79b0c49d96854b1a8f43e149053ca325, State: Running, Role: LEADER
I20240502 01:52:33.812361 306 leader_election.cc:302] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79b0c49d96854b1a8f43e149053ca325, 968c06ee65964aceb131114dfbd683de; no voters:
I20240502 01:52:33.813658 443 raft_consensus.cc:482] T d1a537af86f146aeb6597e1d61274cf0 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:33.813562 444 consensus_queue.cc:237] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.813800 443 raft_consensus.cc:504] T d1a537af86f146aeb6597e1d61274cf0 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.814415 443 leader_election.cc:288] T d1a537af86f146aeb6597e1d61274cf0 P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847), 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451)
I20240502 01:52:33.815618 439 ts_tablet_manager.cc:1398] T b9f486c135914f558f3c847ac02dda8a P 968c06ee65964aceb131114dfbd683de: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:52:33.815753 32709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf049ada294349e3b3811e1397d4851d" candidate_uuid: "79b0c49d96854b1a8f43e149053ca325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5"
I20240502 01:52:33.816020 32709 raft_consensus.cc:3037] T cf049ada294349e3b3811e1397d4851d P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.816787 439 tablet_bootstrap.cc:492] T f95e1c9c66a44807a89302c6c53d90f2 P 968c06ee65964aceb131114dfbd683de: Bootstrap starting.
I20240502 01:52:33.818840 32709 raft_consensus.cc:2454] T cf049ada294349e3b3811e1397d4851d P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79b0c49d96854b1a8f43e149053ca325 in term 1.
I20240502 01:52:33.819077 479 raft_consensus.cc:2781] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:33.819365 479 raft_consensus.cc:686] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de [term 1 LEADER]: Becoming Leader. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Running, Role: LEADER
I20240502 01:52:33.819525 439 tablet_bootstrap.cc:654] T f95e1c9c66a44807a89302c6c53d90f2 P 968c06ee65964aceb131114dfbd683de: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.820004 479 consensus_queue.cc:237] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.822687 306 leader_election.cc:302] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79b0c49d96854b1a8f43e149053ca325, 968c06ee65964aceb131114dfbd683de; no voters:
I20240502 01:52:33.823000 447 raft_consensus.cc:2781] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:33.825242 443 raft_consensus.cc:482] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:33.830736 498 raft_consensus.cc:482] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:33.830756 443 raft_consensus.cc:504] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.831107 32511 leader_election.cc:302] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 79b0c49d96854b1a8f43e149053ca325; no voters:
I20240502 01:52:33.831382 443 leader_election.cc:288] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847), 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451)
W20240502 01:52:33.831964 32512 leader_election.cc:341] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:33.832265 498 raft_consensus.cc:504] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.835212 444 raft_consensus.cc:2781] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:33.835381 444 raft_consensus.cc:686] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325 [term 1 LEADER]: Becoming Leader. State: Replica: 79b0c49d96854b1a8f43e149053ca325, State: Running, Role: LEADER
I20240502 01:52:33.835650 444 consensus_queue.cc:237] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.836649 32578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657f6ad36cf34cbcaca11cd331672993" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325" is_pre_election: true
I20240502 01:52:33.836902 32578 raft_consensus.cc:2454] T 657f6ad36cf34cbcaca11cd331672993 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50fc42d44d3441eaa2616635544066a5 in term 0.
I20240502 01:52:33.837615 32578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf049ada294349e3b3811e1397d4851d" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325" is_pre_election: true
I20240502 01:52:33.838176 32577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a537af86f146aeb6597e1d61274cf0" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325" is_pre_election: true
I20240502 01:52:33.838760 32576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79527d85ba284c04ae435ed71a3d2d0c" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325" is_pre_election: true
I20240502 01:52:33.839555 32576 raft_consensus.cc:2454] T 79527d85ba284c04ae435ed71a3d2d0c P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50fc42d44d3441eaa2616635544066a5 in term 0.
I20240502 01:52:33.839742 447 raft_consensus.cc:686] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de [term 1 LEADER]: Becoming Leader. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Running, Role: LEADER
I20240502 01:52:33.840058 447 consensus_queue.cc:237] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.842746 438 tablet_bootstrap.cc:492] T 9e11386999d543a99ca5b08059e3048f P 79b0c49d96854b1a8f43e149053ca325: No bootstrap required, opened a new log
I20240502 01:52:33.842962 438 ts_tablet_manager.cc:1367] T 9e11386999d543a99ca5b08059e3048f P 79b0c49d96854b1a8f43e149053ca325: Time spent bootstrapping tablet: real 0.088s user 0.022s sys 0.016s
I20240502 01:52:33.842890 32435 catalog_manager.cc:5484] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de reported cstate change: term changed from 0 to 1, leader changed from <none> to 968c06ee65964aceb131114dfbd683de (127.31.161.195). New cstate: current_term: 1 leader_uuid: "968c06ee65964aceb131114dfbd683de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:33.844297 438 raft_consensus.cc:348] T 9e11386999d543a99ca5b08059e3048f P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.839092 32575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f486c135914f558f3c847ac02dda8a" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325" is_pre_election: true
I20240502 01:52:33.844517 438 raft_consensus.cc:374] T 9e11386999d543a99ca5b08059e3048f P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.844614 32575 raft_consensus.cc:2454] T b9f486c135914f558f3c847ac02dda8a P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50fc42d44d3441eaa2616635544066a5 in term 0.
I20240502 01:52:33.844648 438 raft_consensus.cc:729] T 9e11386999d543a99ca5b08059e3048f P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b0c49d96854b1a8f43e149053ca325, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.844828 438 consensus_queue.cc:260] T 9e11386999d543a99ca5b08059e3048f P 79b0c49d96854b1a8f43e149053ca325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.845417 438 ts_tablet_manager.cc:1398] T 9e11386999d543a99ca5b08059e3048f P 79b0c49d96854b1a8f43e149053ca325: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:52:33.845690 438 tablet_bootstrap.cc:492] T d1a537af86f146aeb6597e1d61274cf0 P 79b0c49d96854b1a8f43e149053ca325: Bootstrap starting.
I20240502 01:52:33.845224 32434 catalog_manager.cc:5484] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193). New cstate: current_term: 1 leader_uuid: "79b0c49d96854b1a8f43e149053ca325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:33.846514 372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657f6ad36cf34cbcaca11cd331672993" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c06ee65964aceb131114dfbd683de" is_pre_election: true
I20240502 01:52:33.847278 372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf049ada294349e3b3811e1397d4851d" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c06ee65964aceb131114dfbd683de" is_pre_election: true
I20240502 01:52:33.848035 372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a537af86f146aeb6597e1d61274cf0" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c06ee65964aceb131114dfbd683de" is_pre_election: true
I20240502 01:52:33.848816 372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79527d85ba284c04ae435ed71a3d2d0c" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c06ee65964aceb131114dfbd683de" is_pre_election: true
I20240502 01:52:33.849050 372 raft_consensus.cc:2454] T 79527d85ba284c04ae435ed71a3d2d0c P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50fc42d44d3441eaa2616635544066a5 in term 0.
I20240502 01:52:33.851016 439 tablet_bootstrap.cc:492] T f95e1c9c66a44807a89302c6c53d90f2 P 968c06ee65964aceb131114dfbd683de: No bootstrap required, opened a new log
I20240502 01:52:33.851199 439 ts_tablet_manager.cc:1367] T f95e1c9c66a44807a89302c6c53d90f2 P 968c06ee65964aceb131114dfbd683de: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.006s
I20240502 01:52:33.851363 32435 catalog_manager.cc:5484] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de reported cstate change: term changed from 0 to 1, leader changed from <none> to 968c06ee65964aceb131114dfbd683de (127.31.161.195). New cstate: current_term: 1 leader_uuid: "968c06ee65964aceb131114dfbd683de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:33.853242 32709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a40253e73f40089885747aa6c4c635" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5" is_pre_election: true
I20240502 01:52:33.853591 32709 raft_consensus.cc:2379] T 91a40253e73f40089885747aa6c4c635 P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 968c06ee65964aceb131114dfbd683de in current term 1: Already voted for candidate 79b0c49d96854b1a8f43e149053ca325 in this term.
I20240502 01:52:33.854262 32709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a153b9fa8359422eb28faa8f573edb58" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5" is_pre_election: true
I20240502 01:52:33.854475 32709 raft_consensus.cc:2454] T a153b9fa8359422eb28faa8f573edb58 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 0.
I20240502 01:52:33.852159 439 raft_consensus.cc:348] T f95e1c9c66a44807a89302c6c53d90f2 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.854894 439 raft_consensus.cc:374] T f95e1c9c66a44807a89302c6c53d90f2 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.854992 439 raft_consensus.cc:729] T f95e1c9c66a44807a89302c6c53d90f2 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.852721 372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f486c135914f558f3c847ac02dda8a" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c06ee65964aceb131114dfbd683de" is_pre_election: true
I20240502 01:52:33.855381 372 raft_consensus.cc:2454] T b9f486c135914f558f3c847ac02dda8a P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50fc42d44d3441eaa2616635544066a5 in term 0.
I20240502 01:52:33.855242 439 consensus_queue.cc:260] T f95e1c9c66a44807a89302c6c53d90f2 P 968c06ee65964aceb131114dfbd683de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.855867 438 tablet_bootstrap.cc:654] T d1a537af86f146aeb6597e1d61274cf0 P 79b0c49d96854b1a8f43e149053ca325: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.856020 439 ts_tablet_manager.cc:1398] T f95e1c9c66a44807a89302c6c53d90f2 P 968c06ee65964aceb131114dfbd683de: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:52:33.856302 439 tablet_bootstrap.cc:492] T 657f6ad36cf34cbcaca11cd331672993 P 968c06ee65964aceb131114dfbd683de: Bootstrap starting.
I20240502 01:52:33.856724 443 raft_consensus.cc:482] T 06e288c44ec5454caade0f9fd5b86822 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:33.856863 443 raft_consensus.cc:504] T 06e288c44ec5454caade0f9fd5b86822 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.857535 443 leader_election.cc:288] T 06e288c44ec5454caade0f9fd5b86822 P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847), 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451)
I20240502 01:52:33.857997 439 tablet_bootstrap.cc:654] T 657f6ad36cf34cbcaca11cd331672993 P 968c06ee65964aceb131114dfbd683de: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.859968 32707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd7160232773423590368d00a1cb8e88" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5" is_pre_election: true
I20240502 01:52:33.860174 32708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10898226a6644583915e21ce167afbed" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5" is_pre_election: true
I20240502 01:52:33.860297 32709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e11386999d543a99ca5b08059e3048f" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5" is_pre_election: true
I20240502 01:52:33.860394 32708 raft_consensus.cc:2454] T 10898226a6644583915e21ce167afbed P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 0.
I20240502 01:52:33.860467 32709 raft_consensus.cc:2454] T 9e11386999d543a99ca5b08059e3048f P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 0.
I20240502 01:52:33.860196 32707 raft_consensus.cc:2454] T fd7160232773423590368d00a1cb8e88 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 0.
I20240502 01:52:33.860991 32707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd7160232773423590368d00a1cb8e88" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5"
I20240502 01:52:33.861111 32709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10898226a6644583915e21ce167afbed" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5"
I20240502 01:52:33.861214 32707 raft_consensus.cc:3037] T fd7160232773423590368d00a1cb8e88 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.861316 32709 raft_consensus.cc:3037] T 10898226a6644583915e21ce167afbed P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.861351 32434 catalog_manager.cc:5484] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193). New cstate: current_term: 1 leader_uuid: "79b0c49d96854b1a8f43e149053ca325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:33.861897 32434 catalog_manager.cc:5484] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193). New cstate: current_term: 1 leader_uuid: "79b0c49d96854b1a8f43e149053ca325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:33.862331 32434 catalog_manager.cc:5484] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325 reported cstate change: term changed from 0 to 1, leader changed from <none> to 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193). New cstate: current_term: 1 leader_uuid: "79b0c49d96854b1a8f43e149053ca325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:33.864112 32707 raft_consensus.cc:2454] T fd7160232773423590368d00a1cb8e88 P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 1.
I20240502 01:52:33.864197 32709 raft_consensus.cc:2454] T 10898226a6644583915e21ce167afbed P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 1.
I20240502 01:52:33.864464 32643 leader_election.cc:302] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 79b0c49d96854b1a8f43e149053ca325; no voters:
I20240502 01:52:33.865129 443 raft_consensus.cc:2781] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:33.864720 498 leader_election.cc:288] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847), 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451)
I20240502 01:52:33.865252 443 raft_consensus.cc:482] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:33.865350 443 raft_consensus.cc:3037] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:52:33.865499 32643 leader_election.cc:341] T d1a537af86f146aeb6597e1d61274cf0 P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:33.865707 439 tablet_bootstrap.cc:492] T 657f6ad36cf34cbcaca11cd331672993 P 968c06ee65964aceb131114dfbd683de: No bootstrap required, opened a new log
I20240502 01:52:33.865885 439 ts_tablet_manager.cc:1367] T 657f6ad36cf34cbcaca11cd331672993 P 968c06ee65964aceb131114dfbd683de: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 01:52:33.865921 32643 leader_election.cc:302] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 79b0c49d96854b1a8f43e149053ca325; no voters:
I20240502 01:52:33.866282 32643 leader_election.cc:302] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 79b0c49d96854b1a8f43e149053ca325; no voters:
I20240502 01:52:33.866659 498 raft_consensus.cc:2781] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:33.866581 439 raft_consensus.cc:348] T 657f6ad36cf34cbcaca11cd331672993 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } }
I20240502 01:52:33.866721 447 raft_consensus.cc:482] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:33.866830 439 raft_consensus.cc:374] T 657f6ad36cf34cbcaca11cd331672993 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.866886 498 raft_consensus.cc:482] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:33.866868 447 raft_consensus.cc:504] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.866925 439 raft_consensus.cc:729] T 657f6ad36cf34cbcaca11cd331672993 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.867142 498 raft_consensus.cc:3037] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.867197 439 consensus_queue.cc:260] T 657f6ad36cf34cbcaca11cd331672993 P 968c06ee65964aceb131114dfbd683de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } }
I20240502 01:52:33.867470 447 leader_election.cc:288] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619), 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847)
I20240502 01:52:33.867877 439 ts_tablet_manager.cc:1398] T 657f6ad36cf34cbcaca11cd331672993 P 968c06ee65964aceb131114dfbd683de: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:52:33.868103 443 raft_consensus.cc:504] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } }
I20240502 01:52:33.868240 439 tablet_bootstrap.cc:492] T cf049ada294349e3b3811e1397d4851d P 968c06ee65964aceb131114dfbd683de: Bootstrap starting.
I20240502 01:52:33.868211 32578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e288c44ec5454caade0f9fd5b86822" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325" is_pre_election: true
W20240502 01:52:33.868705 306 leader_election.cc:341] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:33.868760 443 leader_election.cc:288] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 election: Requested vote from peers 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451), 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847)
I20240502 01:52:33.868933 443 raft_consensus.cc:2781] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:33.869027 443 raft_consensus.cc:482] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:33.869148 443 raft_consensus.cc:3037] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.869727 439 tablet_bootstrap.cc:654] T cf049ada294349e3b3811e1397d4851d P 968c06ee65964aceb131114dfbd683de: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.871078 437 tablet_bootstrap.cc:492] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 50fc42d44d3441eaa2616635544066a5: No bootstrap required, opened a new log
I20240502 01:52:33.871251 437 ts_tablet_manager.cc:1367] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 50fc42d44d3441eaa2616635544066a5: Time spent bootstrapping tablet: real 0.064s user 0.004s sys 0.017s
I20240502 01:52:33.871762 439 tablet_bootstrap.cc:492] T cf049ada294349e3b3811e1397d4851d P 968c06ee65964aceb131114dfbd683de: No bootstrap required, opened a new log
I20240502 01:52:33.871676 443 raft_consensus.cc:504] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.871979 439 ts_tablet_manager.cc:1367] T cf049ada294349e3b3811e1397d4851d P 968c06ee65964aceb131114dfbd683de: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:52:33.872278 443 leader_election.cc:288] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 election: Requested vote from peers 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847), 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451)
I20240502 01:52:33.872418 437 raft_consensus.cc:348] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.872658 437 raft_consensus.cc:374] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.872727 439 raft_consensus.cc:348] T cf049ada294349e3b3811e1397d4851d P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.872962 439 raft_consensus.cc:374] T cf049ada294349e3b3811e1397d4851d P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.873060 439 raft_consensus.cc:729] T cf049ada294349e3b3811e1397d4851d P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.873307 439 consensus_queue.cc:260] T cf049ada294349e3b3811e1397d4851d P 968c06ee65964aceb131114dfbd683de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.873674 437 raft_consensus.cc:729] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fc42d44d3441eaa2616635544066a5, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.873927 437 consensus_queue.cc:260] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 50fc42d44d3441eaa2616635544066a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.874037 447 raft_consensus.cc:482] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:33.874063 439 ts_tablet_manager.cc:1398] T cf049ada294349e3b3811e1397d4851d P 968c06ee65964aceb131114dfbd683de: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:52:33.874202 447 raft_consensus.cc:504] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.874382 439 tablet_bootstrap.cc:492] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de: Bootstrap starting.
I20240502 01:52:33.874886 447 leader_election.cc:288] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847), 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619)
I20240502 01:52:33.874977 437 ts_tablet_manager.cc:1398] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 50fc42d44d3441eaa2616635544066a5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:52:33.875900 437 tablet_bootstrap.cc:492] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5: Bootstrap starting.
I20240502 01:52:33.876224 439 tablet_bootstrap.cc:654] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.876192 32578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba40300b09fa4a7f9d0b1b3eaf238ebc" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325" is_pre_election: true
I20240502 01:52:33.877504 437 tablet_bootstrap.cc:654] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5: Neither blocks nor log segments found. Creating new log.
W20240502 01:52:33.877625 306 leader_election.cc:341] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:33.873495 498 raft_consensus.cc:504] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.880223 498 leader_election.cc:288] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 election: Requested vote from peers 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847), 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451)
I20240502 01:52:33.880731 438 tablet_bootstrap.cc:492] T d1a537af86f146aeb6597e1d61274cf0 P 79b0c49d96854b1a8f43e149053ca325: No bootstrap required, opened a new log
I20240502 01:52:33.880888 438 ts_tablet_manager.cc:1367] T d1a537af86f146aeb6597e1d61274cf0 P 79b0c49d96854b1a8f43e149053ca325: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.005s
W20240502 01:52:33.881362 32643 leader_election.cc:341] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:33.881592 439 tablet_bootstrap.cc:492] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de: No bootstrap required, opened a new log
I20240502 01:52:33.881779 439 ts_tablet_manager.cc:1367] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.007s
I20240502 01:52:33.881685 438 raft_consensus.cc:348] T d1a537af86f146aeb6597e1d61274cf0 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.881892 438 raft_consensus.cc:374] T d1a537af86f146aeb6597e1d61274cf0 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 01:52:33.881875 32643 leader_election.cc:341] T cf049ada294349e3b3811e1397d4851d P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:33.881973 438 raft_consensus.cc:729] T d1a537af86f146aeb6597e1d61274cf0 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b0c49d96854b1a8f43e149053ca325, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.882001 32643 leader_election.cc:302] T cf049ada294349e3b3811e1397d4851d P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5; no voters: 79b0c49d96854b1a8f43e149053ca325, 968c06ee65964aceb131114dfbd683de
I20240502 01:52:33.882265 498 raft_consensus.cc:2726] T cf049ada294349e3b3811e1397d4851d P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:52:33.882233 438 consensus_queue.cc:260] T d1a537af86f146aeb6597e1d61274cf0 P 79b0c49d96854b1a8f43e149053ca325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
W20240502 01:52:33.882416 32643 leader_election.cc:341] T d1a537af86f146aeb6597e1d61274cf0 P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:33.882531 32643 leader_election.cc:302] T d1a537af86f146aeb6597e1d61274cf0 P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5; no voters: 79b0c49d96854b1a8f43e149053ca325, 968c06ee65964aceb131114dfbd683de
I20240502 01:52:33.882769 498 raft_consensus.cc:2726] T d1a537af86f146aeb6597e1d61274cf0 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:52:33.882861 438 ts_tablet_manager.cc:1398] T d1a537af86f146aeb6597e1d61274cf0 P 79b0c49d96854b1a8f43e149053ca325: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:52:33.883148 438 tablet_bootstrap.cc:492] T 967284a71df1439aa5b27bc48c957c09 P 79b0c49d96854b1a8f43e149053ca325: Bootstrap starting.
I20240502 01:52:33.883297 305 leader_election.cc:302] T 91a40253e73f40089885747aa6c4c635 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 968c06ee65964aceb131114dfbd683de; no voters: 50fc42d44d3441eaa2616635544066a5, 79b0c49d96854b1a8f43e149053ca325
I20240502 01:52:33.883905 305 leader_election.cc:302] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 968c06ee65964aceb131114dfbd683de; no voters: 79b0c49d96854b1a8f43e149053ca325
I20240502 01:52:33.884204 32709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e288c44ec5454caade0f9fd5b86822" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5" is_pre_election: true
I20240502 01:52:33.884418 32709 raft_consensus.cc:2454] T 06e288c44ec5454caade0f9fd5b86822 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 0.
I20240502 01:52:33.884456 305 leader_election.cc:302] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 968c06ee65964aceb131114dfbd683de; no voters: 79b0c49d96854b1a8f43e149053ca325
I20240502 01:52:33.884707 438 tablet_bootstrap.cc:654] T 967284a71df1439aa5b27bc48c957c09 P 79b0c49d96854b1a8f43e149053ca325: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.884987 32709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba40300b09fa4a7f9d0b1b3eaf238ebc" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5" is_pre_election: true
I20240502 01:52:33.885202 32709 raft_consensus.cc:2454] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 0.
I20240502 01:52:33.885514 305 leader_election.cc:302] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 968c06ee65964aceb131114dfbd683de; no voters: 79b0c49d96854b1a8f43e149053ca325
I20240502 01:52:33.885905 32578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e288c44ec5454caade0f9fd5b86822" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325" is_pre_election: true
I20240502 01:52:33.886440 447 raft_consensus.cc:2726] T 91a40253e73f40089885747aa6c4c635 P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:52:33.886637 447 raft_consensus.cc:2781] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:33.886514 439 raft_consensus.cc:348] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.886721 447 raft_consensus.cc:482] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:33.886773 439 raft_consensus.cc:374] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.886842 447 raft_consensus.cc:3037] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.886881 439 raft_consensus.cc:729] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.887101 439 consensus_queue.cc:260] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.887602 439 ts_tablet_manager.cc:1398] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240502 01:52:33.887902 439 tablet_bootstrap.cc:492] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de: Bootstrap starting.
I20240502 01:52:33.889219 479 raft_consensus.cc:2781] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:33.889325 479 raft_consensus.cc:482] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:33.889428 479 raft_consensus.cc:3037] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.889626 439 tablet_bootstrap.cc:654] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de: Neither blocks nor log segments found. Creating new log.
W20240502 01:52:33.889984 32643 leader_election.cc:341] T 06e288c44ec5454caade0f9fd5b86822 P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:33.891469 437 tablet_bootstrap.cc:492] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5: No bootstrap required, opened a new log
I20240502 01:52:33.891597 437 ts_tablet_manager.cc:1367] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.003s
I20240502 01:52:33.891990 479 raft_consensus.cc:504] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.892340 437 raft_consensus.cc:348] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.892545 437 raft_consensus.cc:374] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.892630 437 raft_consensus.cc:729] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50fc42d44d3441eaa2616635544066a5, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.892798 497 raft_consensus.cc:2781] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:33.892918 497 raft_consensus.cc:482] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:33.892845 437 consensus_queue.cc:260] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.893013 497 raft_consensus.cc:3037] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.894816 437 ts_tablet_manager.cc:1398] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:52:33.895918 497 raft_consensus.cc:504] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.896828 32575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657f6ad36cf34cbcaca11cd331672993" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325"
I20240502 01:52:33.897028 32575 raft_consensus.cc:3037] T 657f6ad36cf34cbcaca11cd331672993 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.897644 305 leader_election.cc:302] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 968c06ee65964aceb131114dfbd683de; no voters: 79b0c49d96854b1a8f43e149053ca325
I20240502 01:52:33.898023 508 raft_consensus.cc:2781] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:33.898113 508 raft_consensus.cc:482] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:33.898223 508 raft_consensus.cc:3037] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.900100 32575 raft_consensus.cc:2454] T 657f6ad36cf34cbcaca11cd331672993 P 79b0c49d96854b1a8f43e149053ca325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50fc42d44d3441eaa2616635544066a5 in term 1.
I20240502 01:52:33.900545 32576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f486c135914f558f3c847ac02dda8a" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325"
I20240502 01:52:33.900738 32576 raft_consensus.cc:3037] T b9f486c135914f558f3c847ac02dda8a P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.900852 508 raft_consensus.cc:504] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.902415 439 tablet_bootstrap.cc:492] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de: No bootstrap required, opened a new log
I20240502 01:52:33.902549 439 ts_tablet_manager.cc:1367] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.005s
I20240502 01:52:33.902596 32578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e11386999d543a99ca5b08059e3048f" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325"
I20240502 01:52:33.902771 32578 raft_consensus.cc:3037] T 9e11386999d543a99ca5b08059e3048f P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.903100 369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79527d85ba284c04ae435ed71a3d2d0c" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c06ee65964aceb131114dfbd683de"
I20240502 01:52:33.903357 369 raft_consensus.cc:3037] T 79527d85ba284c04ae435ed71a3d2d0c P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.903416 439 raft_consensus.cc:348] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.903561 32576 raft_consensus.cc:2454] T b9f486c135914f558f3c847ac02dda8a P 79b0c49d96854b1a8f43e149053ca325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50fc42d44d3441eaa2616635544066a5 in term 1.
I20240502 01:52:33.903637 439 raft_consensus.cc:374] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.903729 439 raft_consensus.cc:729] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.903936 439 consensus_queue.cc:260] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.905148 32578 raft_consensus.cc:2454] T 9e11386999d543a99ca5b08059e3048f P 79b0c49d96854b1a8f43e149053ca325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 1.
I20240502 01:52:33.906077 438 tablet_bootstrap.cc:492] T 967284a71df1439aa5b27bc48c957c09 P 79b0c49d96854b1a8f43e149053ca325: No bootstrap required, opened a new log
I20240502 01:52:33.906214 438 ts_tablet_manager.cc:1367] T 967284a71df1439aa5b27bc48c957c09 P 79b0c49d96854b1a8f43e149053ca325: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.005s
I20240502 01:52:33.906244 369 raft_consensus.cc:2454] T 79527d85ba284c04ae435ed71a3d2d0c P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50fc42d44d3441eaa2616635544066a5 in term 1.
I20240502 01:52:33.906491 306 leader_election.cc:302] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79b0c49d96854b1a8f43e149053ca325, 968c06ee65964aceb131114dfbd683de; no voters:
I20240502 01:52:33.906684 370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9f486c135914f558f3c847ac02dda8a" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c06ee65964aceb131114dfbd683de"
I20240502 01:52:33.906980 370 raft_consensus.cc:3037] T b9f486c135914f558f3c847ac02dda8a P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.906939 438 raft_consensus.cc:348] T 967284a71df1439aa5b27bc48c957c09 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.907131 438 raft_consensus.cc:374] T 967284a71df1439aa5b27bc48c957c09 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.907219 438 raft_consensus.cc:729] T 967284a71df1439aa5b27bc48c957c09 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b0c49d96854b1a8f43e149053ca325, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.907435 438 consensus_queue.cc:260] T 967284a71df1439aa5b27bc48c957c09 P 79b0c49d96854b1a8f43e149053ca325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.907987 438 ts_tablet_manager.cc:1398] T 967284a71df1439aa5b27bc48c957c09 P 79b0c49d96854b1a8f43e149053ca325: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:52:33.908284 438 tablet_bootstrap.cc:492] T 06e288c44ec5454caade0f9fd5b86822 P 79b0c49d96854b1a8f43e149053ca325: Bootstrap starting.
I20240502 01:52:33.909291 371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657f6ad36cf34cbcaca11cd331672993" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c06ee65964aceb131114dfbd683de"
I20240502 01:52:33.909580 371 raft_consensus.cc:3037] T 657f6ad36cf34cbcaca11cd331672993 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.910012 438 tablet_bootstrap.cc:654] T 06e288c44ec5454caade0f9fd5b86822 P 79b0c49d96854b1a8f43e149053ca325: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.911547 508 leader_election.cc:288] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 election: Requested vote from peers 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847), 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619)
I20240502 01:52:33.911794 508 raft_consensus.cc:2781] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:33.911945 508 raft_consensus.cc:686] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de [term 1 LEADER]: Becoming Leader. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Running, Role: LEADER
I20240502 01:52:33.912173 508 consensus_queue.cc:237] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.912509 371 raft_consensus.cc:2454] T 657f6ad36cf34cbcaca11cd331672993 P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50fc42d44d3441eaa2616635544066a5 in term 1.
I20240502 01:52:33.912983 372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e288c44ec5454caade0f9fd5b86822" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c06ee65964aceb131114dfbd683de" is_pre_election: true
I20240502 01:52:33.913398 372 raft_consensus.cc:2379] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 50fc42d44d3441eaa2616635544066a5 in current term 1: Already voted for candidate 968c06ee65964aceb131114dfbd683de in this term.
I20240502 01:52:33.913838 32577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79527d85ba284c04ae435ed71a3d2d0c" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325"
I20240502 01:52:33.914072 32577 raft_consensus.cc:3037] T 79527d85ba284c04ae435ed71a3d2d0c P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.919376 370 raft_consensus.cc:2454] T b9f486c135914f558f3c847ac02dda8a P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50fc42d44d3441eaa2616635544066a5 in term 1.
I20240502 01:52:33.922580 447 raft_consensus.cc:504] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.923803 447 leader_election.cc:288] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 election: Requested vote from peers 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619), 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847)
I20240502 01:52:33.924540 479 leader_election.cc:288] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 election: Requested vote from peers 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847), 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619)
I20240502 01:52:33.924867 32577 raft_consensus.cc:2454] T 79527d85ba284c04ae435ed71a3d2d0c P 79b0c49d96854b1a8f43e149053ca325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50fc42d44d3441eaa2616635544066a5 in term 1.
I20240502 01:52:33.925710 497 leader_election.cc:288] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 election: Requested vote from peers 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619), 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847)
I20240502 01:52:33.925738 32709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba40300b09fa4a7f9d0b1b3eaf238ebc" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5"
I20240502 01:52:33.926035 32709 raft_consensus.cc:3037] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.926654 439 ts_tablet_manager.cc:1398] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de: Time spent starting tablet: real 0.024s user 0.003s sys 0.016s
I20240502 01:52:33.927862 510 raft_consensus.cc:482] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:33.928033 510 raft_consensus.cc:504] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.928406 32709 raft_consensus.cc:2454] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 1.
I20240502 01:52:33.928864 510 leader_election.cc:288] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847), 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619)
I20240502 01:52:33.928894 503 raft_consensus.cc:482] T 626b95f7b22d4c278afda157cf50f7cc P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:33.929046 503 raft_consensus.cc:504] T 626b95f7b22d4c278afda157cf50f7cc P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.929576 32709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a153b9fa8359422eb28faa8f573edb58" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5"
I20240502 01:52:33.929769 32709 raft_consensus.cc:3037] T a153b9fa8359422eb28faa8f573edb58 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.929958 503 leader_election.cc:288] T 626b95f7b22d4c278afda157cf50f7cc P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847), 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451)
I20240502 01:52:33.930626 32707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e11386999d543a99ca5b08059e3048f" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5"
I20240502 01:52:33.930820 32707 raft_consensus.cc:3037] T 9e11386999d543a99ca5b08059e3048f P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.932610 32709 raft_consensus.cc:2454] T a153b9fa8359422eb28faa8f573edb58 P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 1.
I20240502 01:52:33.933378 32707 raft_consensus.cc:2454] T 9e11386999d543a99ca5b08059e3048f P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 1.
I20240502 01:52:33.933650 32643 leader_election.cc:302] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 968c06ee65964aceb131114dfbd683de; no voters:
I20240502 01:52:33.933691 32708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "967284a71df1439aa5b27bc48c957c09" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5" is_pre_election: true
I20240502 01:52:33.933856 32708 raft_consensus.cc:2454] T 967284a71df1439aa5b27bc48c957c09 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 0.
I20240502 01:52:33.934201 503 raft_consensus.cc:2781] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:33.934253 32643 leader_election.cc:302] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 968c06ee65964aceb131114dfbd683de; no voters:
I20240502 01:52:33.934379 503 raft_consensus.cc:686] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5 [term 1 LEADER]: Becoming Leader. State: Replica: 50fc42d44d3441eaa2616635544066a5, State: Running, Role: LEADER
I20240502 01:52:33.934481 443 raft_consensus.cc:2781] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:33.934631 443 raft_consensus.cc:686] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5 [term 1 LEADER]: Becoming Leader. State: Replica: 50fc42d44d3441eaa2616635544066a5, State: Running, Role: LEADER
I20240502 01:52:33.934921 443 consensus_queue.cc:237] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } }
I20240502 01:52:33.935117 503 consensus_queue.cc:237] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.934149 32709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e288c44ec5454caade0f9fd5b86822" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5"
I20240502 01:52:33.937215 32709 raft_consensus.cc:3037] T 06e288c44ec5454caade0f9fd5b86822 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.939158 32643 leader_election.cc:302] T 06e288c44ec5454caade0f9fd5b86822 P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5; no voters: 79b0c49d96854b1a8f43e149053ca325, 968c06ee65964aceb131114dfbd683de
I20240502 01:52:33.939633 32643 leader_election.cc:302] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 968c06ee65964aceb131114dfbd683de; no voters:
I20240502 01:52:33.943693 32709 raft_consensus.cc:2454] T 06e288c44ec5454caade0f9fd5b86822 P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 1.
I20240502 01:52:33.944821 498 raft_consensus.cc:2726] T 06e288c44ec5454caade0f9fd5b86822 P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:52:33.945345 498 raft_consensus.cc:2781] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:33.945484 498 raft_consensus.cc:686] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5 [term 1 LEADER]: Becoming Leader. State: Replica: 50fc42d44d3441eaa2616635544066a5, State: Running, Role: LEADER
I20240502 01:52:33.945758 498 consensus_queue.cc:237] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.948450 438 tablet_bootstrap.cc:492] T 06e288c44ec5454caade0f9fd5b86822 P 79b0c49d96854b1a8f43e149053ca325: No bootstrap required, opened a new log
I20240502 01:52:33.948649 438 ts_tablet_manager.cc:1367] T 06e288c44ec5454caade0f9fd5b86822 P 79b0c49d96854b1a8f43e149053ca325: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.013s
I20240502 01:52:33.949574 438 raft_consensus.cc:348] T 06e288c44ec5454caade0f9fd5b86822 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.949860 438 raft_consensus.cc:374] T 06e288c44ec5454caade0f9fd5b86822 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.949937 438 raft_consensus.cc:729] T 06e288c44ec5454caade0f9fd5b86822 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b0c49d96854b1a8f43e149053ca325, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.950184 438 consensus_queue.cc:260] T 06e288c44ec5454caade0f9fd5b86822 P 79b0c49d96854b1a8f43e149053ca325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.950278 305 leader_election.cc:302] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 968c06ee65964aceb131114dfbd683de; no voters:
I20240502 01:52:33.950603 511 raft_consensus.cc:2781] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:33.950728 438 ts_tablet_manager.cc:1398] T 06e288c44ec5454caade0f9fd5b86822 P 79b0c49d96854b1a8f43e149053ca325: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:52:33.950755 511 raft_consensus.cc:686] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de [term 1 LEADER]: Becoming Leader. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Running, Role: LEADER
I20240502 01:52:33.951231 438 tablet_bootstrap.cc:492] T a153b9fa8359422eb28faa8f573edb58 P 79b0c49d96854b1a8f43e149053ca325: Bootstrap starting.
I20240502 01:52:33.951575 305 leader_election.cc:302] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 968c06ee65964aceb131114dfbd683de; no voters:
I20240502 01:52:33.951596 370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626b95f7b22d4c278afda157cf50f7cc" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c06ee65964aceb131114dfbd683de" is_pre_election: true
I20240502 01:52:33.951849 370 raft_consensus.cc:2379] T 626b95f7b22d4c278afda157cf50f7cc P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 50fc42d44d3441eaa2616635544066a5 in current term 1: Already voted for candidate 79b0c49d96854b1a8f43e149053ca325 in this term.
I20240502 01:52:33.951882 305 leader_election.cc:302] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 968c06ee65964aceb131114dfbd683de; no voters:
I20240502 01:52:33.952118 497 raft_consensus.cc:2781] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:33.952209 305 leader_election.cc:302] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 968c06ee65964aceb131114dfbd683de; no voters:
I20240502 01:52:33.952270 497 raft_consensus.cc:686] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de [term 1 LEADER]: Becoming Leader. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Running, Role: LEADER
I20240502 01:52:33.952461 447 raft_consensus.cc:2781] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:33.952407 32575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "626b95f7b22d4c278afda157cf50f7cc" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325" is_pre_election: true
I20240502 01:52:33.952565 447 raft_consensus.cc:482] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:33.952648 447 raft_consensus.cc:3037] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.952538 497 consensus_queue.cc:237] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.952852 438 tablet_bootstrap.cc:654] T a153b9fa8359422eb28faa8f573edb58 P 79b0c49d96854b1a8f43e149053ca325: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.952867 510 raft_consensus.cc:2781] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:33.953043 510 raft_consensus.cc:686] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de [term 1 LEADER]: Becoming Leader. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Running, Role: LEADER
I20240502 01:52:33.953368 510 consensus_queue.cc:237] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.954629 32643 leader_election.cc:302] T 626b95f7b22d4c278afda157cf50f7cc P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5; no voters: 79b0c49d96854b1a8f43e149053ca325, 968c06ee65964aceb131114dfbd683de
I20240502 01:52:33.954912 503 raft_consensus.cc:3037] T 626b95f7b22d4c278afda157cf50f7cc P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.957849 503 raft_consensus.cc:2726] T 626b95f7b22d4c278afda157cf50f7cc P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:52:33.959244 32435 catalog_manager.cc:5484] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50fc42d44d3441eaa2616635544066a5 (127.31.161.194). New cstate: current_term: 1 leader_uuid: "50fc42d44d3441eaa2616635544066a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:33.961835 447 raft_consensus.cc:504] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.962455 447 leader_election.cc:288] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 election: Requested vote from peers 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847), 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619)
I20240502 01:52:33.951087 511 consensus_queue.cc:237] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.963120 32434 catalog_manager.cc:5484] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de reported cstate change: term changed from 0 to 1, leader changed from <none> to 968c06ee65964aceb131114dfbd683de (127.31.161.195). New cstate: current_term: 1 leader_uuid: "968c06ee65964aceb131114dfbd683de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:33.965579 32709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "967284a71df1439aa5b27bc48c957c09" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5"
I20240502 01:52:33.965772 32709 raft_consensus.cc:3037] T 967284a71df1439aa5b27bc48c957c09 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:52:33.966408 306 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.161.193:35847, user_credentials={real_user=slave}} blocked reactor thread for 58553.5us
I20240502 01:52:33.967335 32575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba40300b09fa4a7f9d0b1b3eaf238ebc" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325"
I20240502 01:52:33.968323 32709 raft_consensus.cc:2454] T 967284a71df1439aa5b27bc48c957c09 P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 1.
I20240502 01:52:33.968436 32575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a153b9fa8359422eb28faa8f573edb58" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325"
I20240502 01:52:33.968731 305 leader_election.cc:302] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 968c06ee65964aceb131114dfbd683de; no voters:
I20240502 01:52:33.969028 511 raft_consensus.cc:2781] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:33.969269 511 raft_consensus.cc:686] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de [term 1 LEADER]: Becoming Leader. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Running, Role: LEADER
I20240502 01:52:33.969399 32575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06e288c44ec5454caade0f9fd5b86822" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325"
I20240502 01:52:33.969305 32434 catalog_manager.cc:5484] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de reported cstate change: term changed from 0 to 1, leader changed from <none> to 968c06ee65964aceb131114dfbd683de (127.31.161.195). New cstate: current_term: 1 leader_uuid: "968c06ee65964aceb131114dfbd683de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:33.969601 32575 raft_consensus.cc:3037] T 06e288c44ec5454caade0f9fd5b86822 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.969513 511 consensus_queue.cc:237] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.969838 32434 catalog_manager.cc:5484] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de reported cstate change: term changed from 0 to 1, leader changed from <none> to 968c06ee65964aceb131114dfbd683de (127.31.161.195). New cstate: current_term: 1 leader_uuid: "968c06ee65964aceb131114dfbd683de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:33.970360 32434 catalog_manager.cc:5484] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de reported cstate change: term changed from 0 to 1, leader changed from <none> to 968c06ee65964aceb131114dfbd683de (127.31.161.195). New cstate: current_term: 1 leader_uuid: "968c06ee65964aceb131114dfbd683de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:33.972633 32575 raft_consensus.cc:2454] T 06e288c44ec5454caade0f9fd5b86822 P 79b0c49d96854b1a8f43e149053ca325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 1.
I20240502 01:52:33.973902 32575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "967284a71df1439aa5b27bc48c957c09" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325" is_pre_election: true
I20240502 01:52:33.974220 32575 raft_consensus.cc:2454] T 967284a71df1439aa5b27bc48c957c09 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 0.
I20240502 01:52:33.974956 32575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "967284a71df1439aa5b27bc48c957c09" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325"
I20240502 01:52:33.975150 32575 raft_consensus.cc:3037] T 967284a71df1439aa5b27bc48c957c09 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:33.979525 32434 catalog_manager.cc:5484] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50fc42d44d3441eaa2616635544066a5 (127.31.161.194). New cstate: current_term: 1 leader_uuid: "50fc42d44d3441eaa2616635544066a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:33.980091 32434 catalog_manager.cc:5484] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50fc42d44d3441eaa2616635544066a5 (127.31.161.194). New cstate: current_term: 1 leader_uuid: "50fc42d44d3441eaa2616635544066a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:33.980351 438 tablet_bootstrap.cc:492] T a153b9fa8359422eb28faa8f573edb58 P 79b0c49d96854b1a8f43e149053ca325: No bootstrap required, opened a new log
I20240502 01:52:33.980502 438 ts_tablet_manager.cc:1367] T a153b9fa8359422eb28faa8f573edb58 P 79b0c49d96854b1a8f43e149053ca325: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.016s
I20240502 01:52:33.981199 32575 raft_consensus.cc:2454] T 967284a71df1439aa5b27bc48c957c09 P 79b0c49d96854b1a8f43e149053ca325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 1.
I20240502 01:52:33.981336 438 raft_consensus.cc:348] T a153b9fa8359422eb28faa8f573edb58 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.981519 438 raft_consensus.cc:374] T a153b9fa8359422eb28faa8f573edb58 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.981602 438 raft_consensus.cc:729] T a153b9fa8359422eb28faa8f573edb58 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b0c49d96854b1a8f43e149053ca325, State: Initialized, Role: FOLLOWER
W20240502 01:52:33.981766 306 leader_election.cc:341] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:33.981881 438 consensus_queue.cc:260] T a153b9fa8359422eb28faa8f573edb58 P 79b0c49d96854b1a8f43e149053ca325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
W20240502 01:52:33.982147 306 leader_election.cc:341] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:33.984380 438 ts_tablet_manager.cc:1398] T a153b9fa8359422eb28faa8f573edb58 P 79b0c49d96854b1a8f43e149053ca325: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:52:33.984648 438 tablet_bootstrap.cc:492] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 79b0c49d96854b1a8f43e149053ca325: Bootstrap starting.
I20240502 01:52:33.984524 32433 catalog_manager.cc:5484] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de reported cstate change: term changed from 0 to 1, leader changed from <none> to 968c06ee65964aceb131114dfbd683de (127.31.161.195). New cstate: current_term: 1 leader_uuid: "968c06ee65964aceb131114dfbd683de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:33.986099 438 tablet_bootstrap.cc:654] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 79b0c49d96854b1a8f43e149053ca325: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.988144 438 tablet_bootstrap.cc:492] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 79b0c49d96854b1a8f43e149053ca325: No bootstrap required, opened a new log
I20240502 01:52:33.988301 438 ts_tablet_manager.cc:1367] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 79b0c49d96854b1a8f43e149053ca325: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:52:33.988898 438 raft_consensus.cc:348] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.989161 438 raft_consensus.cc:374] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.989259 438 raft_consensus.cc:729] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b0c49d96854b1a8f43e149053ca325, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.989526 438 consensus_queue.cc:260] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 79b0c49d96854b1a8f43e149053ca325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.989982 438 ts_tablet_manager.cc:1398] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 79b0c49d96854b1a8f43e149053ca325: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:52:33.990312 438 tablet_bootstrap.cc:492] T f95e1c9c66a44807a89302c6c53d90f2 P 79b0c49d96854b1a8f43e149053ca325: Bootstrap starting.
I20240502 01:52:33.992435 438 tablet_bootstrap.cc:654] T f95e1c9c66a44807a89302c6c53d90f2 P 79b0c49d96854b1a8f43e149053ca325: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:33.994700 438 tablet_bootstrap.cc:492] T f95e1c9c66a44807a89302c6c53d90f2 P 79b0c49d96854b1a8f43e149053ca325: No bootstrap required, opened a new log
I20240502 01:52:33.994832 438 ts_tablet_manager.cc:1367] T f95e1c9c66a44807a89302c6c53d90f2 P 79b0c49d96854b1a8f43e149053ca325: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240502 01:52:33.995527 438 raft_consensus.cc:348] T f95e1c9c66a44807a89302c6c53d90f2 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.995676 438 raft_consensus.cc:374] T f95e1c9c66a44807a89302c6c53d90f2 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:33.995740 438 raft_consensus.cc:729] T f95e1c9c66a44807a89302c6c53d90f2 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79b0c49d96854b1a8f43e149053ca325, State: Initialized, Role: FOLLOWER
I20240502 01:52:33.995904 438 consensus_queue.cc:260] T f95e1c9c66a44807a89302c6c53d90f2 P 79b0c49d96854b1a8f43e149053ca325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:33.996347 438 ts_tablet_manager.cc:1398] T f95e1c9c66a44807a89302c6c53d90f2 P 79b0c49d96854b1a8f43e149053ca325: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:52:34.118216 511 raft_consensus.cc:482] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:34.118402 511 raft_consensus.cc:504] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:34.119071 511 leader_election.cc:288] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619), 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847)
I20240502 01:52:34.119565 32709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a537af86f146aeb6597e1d61274cf0" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5" is_pre_election: true
I20240502 01:52:34.119673 32575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a537af86f146aeb6597e1d61274cf0" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325" is_pre_election: true
I20240502 01:52:34.119846 32709 raft_consensus.cc:2454] T d1a537af86f146aeb6597e1d61274cf0 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 0.
I20240502 01:52:34.119910 32575 raft_consensus.cc:2454] T d1a537af86f146aeb6597e1d61274cf0 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 0.
I20240502 01:52:34.120261 305 leader_election.cc:302] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 968c06ee65964aceb131114dfbd683de; no voters:
I20240502 01:52:34.120548 511 raft_consensus.cc:2781] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:34.120656 511 raft_consensus.cc:482] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:34.120726 511 raft_consensus.cc:3037] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:34.122627 511 raft_consensus.cc:504] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:34.123168 511 leader_election.cc:288] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 election: Requested vote from peers 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619), 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847)
I20240502 01:52:34.123517 32709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a537af86f146aeb6597e1d61274cf0" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5"
I20240502 01:52:34.123692 32709 raft_consensus.cc:3037] T d1a537af86f146aeb6597e1d61274cf0 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:34.123733 32575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1a537af86f146aeb6597e1d61274cf0" candidate_uuid: "968c06ee65964aceb131114dfbd683de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325"
I20240502 01:52:34.123970 32575 raft_consensus.cc:3037] T d1a537af86f146aeb6597e1d61274cf0 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:34.125710 32709 raft_consensus.cc:2454] T d1a537af86f146aeb6597e1d61274cf0 P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 1.
I20240502 01:52:34.126116 305 leader_election.cc:302] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 968c06ee65964aceb131114dfbd683de; no voters:
I20240502 01:52:34.126333 32575 raft_consensus.cc:2454] T d1a537af86f146aeb6597e1d61274cf0 P 79b0c49d96854b1a8f43e149053ca325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 968c06ee65964aceb131114dfbd683de in term 1.
I20240502 01:52:34.126437 511 raft_consensus.cc:2781] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:34.126600 511 raft_consensus.cc:686] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de [term 1 LEADER]: Becoming Leader. State: Replica: 968c06ee65964aceb131114dfbd683de, State: Running, Role: LEADER
I20240502 01:52:34.126894 511 consensus_queue.cc:237] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:34.129782 32434 catalog_manager.cc:5484] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de reported cstate change: term changed from 0 to 1, leader changed from <none> to 968c06ee65964aceb131114dfbd683de (127.31.161.195). New cstate: current_term: 1 leader_uuid: "968c06ee65964aceb131114dfbd683de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:34.189239 444 consensus_queue.cc:1035] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.195297 488 consensus_queue.cc:1035] T 91a40253e73f40089885747aa6c4c635 P 79b0c49d96854b1a8f43e149053ca325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.199756 488 raft_consensus.cc:482] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:34.199891 488 raft_consensus.cc:504] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:34.200471 488 leader_election.cc:288] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 79b0c49d96854b1a8f43e149053ca325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619), 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451)
I20240502 01:52:34.200781 32709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba40300b09fa4a7f9d0b1b3eaf238ebc" candidate_uuid: "79b0c49d96854b1a8f43e149053ca325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50fc42d44d3441eaa2616635544066a5" is_pre_election: true
I20240502 01:52:34.201027 32709 raft_consensus.cc:2379] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 79b0c49d96854b1a8f43e149053ca325 in current term 1: Already voted for candidate 968c06ee65964aceb131114dfbd683de in this term.
I20240502 01:52:34.200954 370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba40300b09fa4a7f9d0b1b3eaf238ebc" candidate_uuid: "79b0c49d96854b1a8f43e149053ca325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c06ee65964aceb131114dfbd683de" is_pre_election: true
I20240502 01:52:34.201642 32512 leader_election.cc:302] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 79b0c49d96854b1a8f43e149053ca325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 79b0c49d96854b1a8f43e149053ca325; no voters: 50fc42d44d3441eaa2616635544066a5, 968c06ee65964aceb131114dfbd683de
I20240502 01:52:34.201921 488 raft_consensus.cc:3037] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:34.203826 488 raft_consensus.cc:2726] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 79b0c49d96854b1a8f43e149053ca325 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:52:34.211350 444 consensus_queue.cc:1035] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.211628 488 consensus_queue.cc:1035] T 29f468ee49544781b84c22fb48f57ede P 79b0c49d96854b1a8f43e149053ca325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.260840 511 consensus_queue.cc:1035] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de [LEADER]: Connected to new peer: Peer: permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.266211 511 consensus_queue.cc:1035] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de [LEADER]: Connected to new peer: Peer: permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.294129 498 raft_consensus.cc:482] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:34.294298 498 raft_consensus.cc:504] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:34.294893 498 leader_election.cc:288] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847), 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451)
I20240502 01:52:34.295462 32575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95e1c9c66a44807a89302c6c53d90f2" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325" is_pre_election: true
I20240502 01:52:34.295557 370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95e1c9c66a44807a89302c6c53d90f2" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c06ee65964aceb131114dfbd683de" is_pre_election: true
I20240502 01:52:34.295738 32575 raft_consensus.cc:2454] T f95e1c9c66a44807a89302c6c53d90f2 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50fc42d44d3441eaa2616635544066a5 in term 0.
I20240502 01:52:34.295789 370 raft_consensus.cc:2454] T f95e1c9c66a44807a89302c6c53d90f2 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50fc42d44d3441eaa2616635544066a5 in term 0.
I20240502 01:52:34.296124 32643 leader_election.cc:302] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 79b0c49d96854b1a8f43e149053ca325; no voters:
I20240502 01:52:34.296357 498 raft_consensus.cc:2781] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:34.296473 498 raft_consensus.cc:482] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:34.296561 498 raft_consensus.cc:3037] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:34.298465 498 raft_consensus.cc:504] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:34.299119 498 leader_election.cc:288] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 election: Requested vote from peers 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847), 968c06ee65964aceb131114dfbd683de (127.31.161.195:38451)
I20240502 01:52:34.299420 32575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95e1c9c66a44807a89302c6c53d90f2" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79b0c49d96854b1a8f43e149053ca325"
I20240502 01:52:34.299669 32575 raft_consensus.cc:3037] T f95e1c9c66a44807a89302c6c53d90f2 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:34.299752 370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f95e1c9c66a44807a89302c6c53d90f2" candidate_uuid: "50fc42d44d3441eaa2616635544066a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "968c06ee65964aceb131114dfbd683de"
I20240502 01:52:34.299911 370 raft_consensus.cc:3037] T f95e1c9c66a44807a89302c6c53d90f2 P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:34.301964 32575 raft_consensus.cc:2454] T f95e1c9c66a44807a89302c6c53d90f2 P 79b0c49d96854b1a8f43e149053ca325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50fc42d44d3441eaa2616635544066a5 in term 1.
I20240502 01:52:34.301970 370 raft_consensus.cc:2454] T f95e1c9c66a44807a89302c6c53d90f2 P 968c06ee65964aceb131114dfbd683de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50fc42d44d3441eaa2616635544066a5 in term 1.
I20240502 01:52:34.302434 32643 leader_election.cc:302] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50fc42d44d3441eaa2616635544066a5, 79b0c49d96854b1a8f43e149053ca325; no voters:
I20240502 01:52:34.302716 498 raft_consensus.cc:2781] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:34.302909 498 raft_consensus.cc:686] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5 [term 1 LEADER]: Becoming Leader. State: Replica: 50fc42d44d3441eaa2616635544066a5, State: Running, Role: LEADER
I20240502 01:52:34.303056 488 consensus_queue.cc:1035] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.303202 498 consensus_queue.cc:237] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } }
I20240502 01:52:34.307410 488 consensus_queue.cc:1035] T 626b95f7b22d4c278afda157cf50f7cc P 79b0c49d96854b1a8f43e149053ca325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.307425 32433 catalog_manager.cc:5484] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50fc42d44d3441eaa2616635544066a5 (127.31.161.194). New cstate: current_term: 1 leader_uuid: "50fc42d44d3441eaa2616635544066a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:34.317452 370 raft_consensus.cc:3037] T cf049ada294349e3b3811e1397d4851d P 968c06ee65964aceb131114dfbd683de [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:34.319972 488 consensus_queue.cc:1035] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.324162 489 consensus_queue.cc:1035] T cf049ada294349e3b3811e1397d4851d P 79b0c49d96854b1a8f43e149053ca325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.340946 511 consensus_queue.cc:1035] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de [LEADER]: Connected to new peer: Peer: permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.346086 510 consensus_queue.cc:1035] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de [LEADER]: Connected to new peer: Peer: permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.370095 32575 raft_consensus.cc:3037] T a153b9fa8359422eb28faa8f573edb58 P 79b0c49d96854b1a8f43e149053ca325 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:34.372797 510 consensus_queue.cc:1035] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de [LEADER]: Connected to new peer: Peer: permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.373801 510 consensus_queue.cc:1035] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de [LEADER]: Connected to new peer: Peer: permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.377324 511 consensus_queue.cc:1035] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de [LEADER]: Connected to new peer: Peer: permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.378531 510 consensus_queue.cc:1035] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de [LEADER]: Connected to new peer: Peer: permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.384440 498 consensus_queue.cc:1035] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.392665 508 consensus_queue.cc:1035] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de [LEADER]: Connected to new peer: Peer: permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.396901 510 consensus_queue.cc:1035] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de [LEADER]: Connected to new peer: Peer: permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.406769 498 consensus_queue.cc:1035] T b9f486c135914f558f3c847ac02dda8a P 50fc42d44d3441eaa2616635544066a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.426280 443 consensus_queue.cc:1035] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.431792 516 consensus_queue.cc:1035] T 657f6ad36cf34cbcaca11cd331672993 P 50fc42d44d3441eaa2616635544066a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.440006 510 consensus_queue.cc:1035] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de [LEADER]: Connected to new peer: Peer: permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.443722 511 consensus_queue.cc:1035] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de [LEADER]: Connected to new peer: Peer: permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.444151 511 consensus_queue.cc:1035] T ba40300b09fa4a7f9d0b1b3eaf238ebc P 968c06ee65964aceb131114dfbd683de [LEADER]: Connected to new peer: Peer: permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.448172 508 consensus_queue.cc:1035] T 967284a71df1439aa5b27bc48c957c09 P 968c06ee65964aceb131114dfbd683de [LEADER]: Connected to new peer: Peer: permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.456069 498 consensus_queue.cc:1035] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.480098 516 consensus_queue.cc:1035] T 79527d85ba284c04ae435ed71a3d2d0c P 50fc42d44d3441eaa2616635544066a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.604156 508 consensus_queue.cc:1035] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de [LEADER]: Connected to new peer: Peer: permanent_uuid: "50fc42d44d3441eaa2616635544066a5" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.611807 508 consensus_queue.cc:1035] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de [LEADER]: Connected to new peer: Peer: permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.806365 443 consensus_queue.cc:1035] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "968c06ee65964aceb131114dfbd683de" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 38451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:34.812428 443 consensus_queue.cc:1035] T f95e1c9c66a44807a89302c6c53d90f2 P 50fc42d44d3441eaa2616635544066a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79b0c49d96854b1a8f43e149053ca325" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:52:34.876857 560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:34.877199 560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:34.885058 560 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 390.675 ms
time per row: 0.195337 ms
I20240502 01:52:35.902874 32687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:52:35.907927 345 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:52:35.917326 32545 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:52:36.074203 32687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:52:36.077325 32545 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:52:36.077741 345 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:52:36.266628 32687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:52:36.267143 32545 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:52:36.267840 345 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:52:36.889204 32687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:52:36.896410 345 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:52:36.896682 32545 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:52:37.067343 32687 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:52:37.068661 32545 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:52:37.072160 345 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:52:37.221901 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 32496
I20240502 01:52:37.228394 32619 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:37.352964 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 32496
W20240502 01:52:37.371042 306 connection.cc:535] server connection from 127.31.161.193:51359 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:52:37.371688 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 32627
I20240502 01:52:37.377508 32750 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:52:37.448597 306 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.161.193:35847: connect: Connection refused (error 111)
W20240502 01:52:37.450063 306 consensus_peers.cc:485] T a153b9fa8359422eb28faa8f573edb58 P 968c06ee65964aceb131114dfbd683de -> Peer 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847): Couldn't send request to peer 79b0c49d96854b1a8f43e149053ca325. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.193:35847: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:37.500845 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 32627
W20240502 01:52:37.520084 305 connection.cc:535] client connection to 127.31.161.194:42619 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:52:37.520299 307 connection.cc:535] server connection from 127.31.161.194:36161 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:52:37.520463 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 32758
W20240502 01:52:37.521668 305 consensus_peers.cc:485] T 10898226a6644583915e21ce167afbed P 968c06ee65964aceb131114dfbd683de -> Peer 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619): Couldn't send request to peer 50fc42d44d3441eaa2616635544066a5. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.194:42619: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:37.526026 412 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:52:37.651979 306 consensus_peers.cc:485] T fd7160232773423590368d00a1cb8e88 P 968c06ee65964aceb131114dfbd683de -> Peer 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847): Couldn't send request to peer 79b0c49d96854b1a8f43e149053ca325. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.193:35847: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:52:37.652227 306 consensus_peers.cc:485] T d1a537af86f146aeb6597e1d61274cf0 P 968c06ee65964aceb131114dfbd683de -> Peer 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847): Couldn't send request to peer 79b0c49d96854b1a8f43e149053ca325. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.193:35847: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:52:37.652364 306 consensus_peers.cc:485] T 9e11386999d543a99ca5b08059e3048f P 968c06ee65964aceb131114dfbd683de -> Peer 79b0c49d96854b1a8f43e149053ca325 (127.31.161.193:35847): Couldn't send request to peer 79b0c49d96854b1a8f43e149053ca325. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.193:35847: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:52:37.652784 305 consensus_peers.cc:485] T 06e288c44ec5454caade0f9fd5b86822 P 968c06ee65964aceb131114dfbd683de -> Peer 50fc42d44d3441eaa2616635544066a5 (127.31.161.194:42619): Couldn't send request to peer 50fc42d44d3441eaa2616635544066a5. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.194:42619: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:37.652859 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 32758
I20240502 01:52:37.670744 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 32404
I20240502 01:52:37.671828 32466 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:37.790347 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 32404
2024-05-02T01:52:37Z chronyd exiting
[ OK ] TableRangeRebalancingTest.Basic (10173 ms)
[----------] 1 test from TableRangeRebalancingTest (10173 ms total)
[----------] 1 test from RebalanceStartSafetyTest
[ RUN ] RebalanceStartSafetyTest.TooManyIgnoredTservers/1
I20240502 01:52:37.838243 32391 test_util.cc:268] Using random seed: -611234420
I20240502 01:52:37.840003 32391 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:52:37.840083 32391 ts_itest-base.cc:113] --------------
I20240502 01:52:37.840123 32391 ts_itest-base.cc:114] 5 tablet servers
I20240502 01:52:37.840163 32391 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:52:37.840209 32391 ts_itest-base.cc:116] --------------
2024-05-02T01:52:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:52:37Z Disabled control of system clock
I20240502 01:52:37.861394 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.161.254:42383
--webserver_interface=127.31.161.254
--webserver_port=0
--builtin_ntp_servers=127.31.161.212:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.161.254:42383
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 01:52:38.183912 808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:38.184260 808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:38.184405 808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:38.191962 808 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:38.192111 808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:52:38.192198 808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:38.192296 808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:52:38.192373 808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:52:38.201687 808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.31.161.254:42383
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.161.254:42383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.31.161.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:38.202993 808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:38.204681 808 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:38.214717 813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:38.215169 814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:38.215448 816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:38.215915 808 server_base.cc:1034] running on GCE node
I20240502 01:52:38.216596 808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:38.217933 808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:38.219136 808 hybrid_clock.cc:648] HybridClock initialized: now 1714614758219093 us; error 35 us; skew 500 ppm
I20240502 01:52:38.219451 808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:38.221891 808 webserver.cc:460] Webserver started at http://127.31.161.254:37419/ using document root <none> and password file <none>
I20240502 01:52:38.222659 808 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:38.222797 808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:38.223140 808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:38.225831 808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "819d28e6ff0449fc95049e7f3d20b121"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-06l8"
I20240502 01:52:38.226631 808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "819d28e6ff0449fc95049e7f3d20b121"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-06l8"
I20240502 01:52:38.232017 808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 01:52:38.235518 822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:38.236785 808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:52:38.236953 808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
uuid: "819d28e6ff0449fc95049e7f3d20b121"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-06l8"
I20240502 01:52:38.237164 808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:38.253772 808 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:38.254611 808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:38.254878 808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:38.271545 808 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.254:42383
I20240502 01:52:38.271565 873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.254:42383 every 8 connection(s)
I20240502 01:52:38.272812 808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:52:38.276374 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 808
I20240502 01:52:38.276588 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:52:38.277321 874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:38.289516 874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121: Bootstrap starting.
I20240502 01:52:38.292412 874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:38.293403 874 log.cc:824] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:38.295713 874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121: No bootstrap required, opened a new log
I20240502 01:52:38.301839 874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819d28e6ff0449fc95049e7f3d20b121" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 42383 } }
I20240502 01:52:38.302175 874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:38.302282 874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 819d28e6ff0449fc95049e7f3d20b121, State: Initialized, Role: FOLLOWER
I20240502 01:52:38.302862 874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819d28e6ff0449fc95049e7f3d20b121" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 42383 } }
I20240502 01:52:38.303053 874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:38.303174 874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:38.303333 874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:38.374526 874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819d28e6ff0449fc95049e7f3d20b121" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 42383 } }
I20240502 01:52:38.375366 874 leader_election.cc:302] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 819d28e6ff0449fc95049e7f3d20b121; no voters:
I20240502 01:52:38.376147 874 leader_election.cc:288] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:38.376261 879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:38.377024 879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121 [term 1 LEADER]: Becoming Leader. State: Replica: 819d28e6ff0449fc95049e7f3d20b121, State: Running, Role: LEADER
I20240502 01:52:38.377624 879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819d28e6ff0449fc95049e7f3d20b121" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 42383 } }
I20240502 01:52:38.378659 874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:52:38.380087 881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 819d28e6ff0449fc95049e7f3d20b121. Latest consensus state: current_term: 1 leader_uuid: "819d28e6ff0449fc95049e7f3d20b121" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819d28e6ff0449fc95049e7f3d20b121" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 42383 } } }
I20240502 01:52:38.380415 881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:38.382220 880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "819d28e6ff0449fc95049e7f3d20b121" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "819d28e6ff0449fc95049e7f3d20b121" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 42383 } } }
I20240502 01:52:38.382615 880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121 [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:38.384083 884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:52:38.389057 884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:52:38.396644 884 catalog_manager.cc:1347] Generated new cluster ID: eefac574d6de4ca99e71841db5b67a96
I20240502 01:52:38.396755 884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:52:38.420732 884 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:52:38.422032 884 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:52:38.432075 884 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 819d28e6ff0449fc95049e7f3d20b121: Generated new TSK 0
I20240502 01:52:38.432843 884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:52:38.439527 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.193:0
--local_ip_for_outbound_sockets=127.31.161.193
--webserver_interface=127.31.161.193
--webserver_port=0
--tserver_master_addrs=127.31.161.254:42383
--builtin_ntp_servers=127.31.161.212:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:52:38.759529 898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:38.759804 898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:38.759932 898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:38.767233 898 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:38.767365 898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:38.767521 898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.193
I20240502 01:52:38.776222 898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.31.161.193
--webserver_port=0
--tserver_master_addrs=127.31.161.254:42383
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.193
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:38.777508 898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:38.779209 898 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:38.790728 903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:38.791116 906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:38.791669 904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:38.792151 898 server_base.cc:1034] running on GCE node
I20240502 01:52:38.792691 898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:38.793582 898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:38.794781 898 hybrid_clock.cc:648] HybridClock initialized: now 1714614758794739 us; error 39 us; skew 500 ppm
I20240502 01:52:38.795070 898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:38.797190 898 webserver.cc:460] Webserver started at http://127.31.161.193:35947/ using document root <none> and password file <none>
I20240502 01:52:38.797868 898 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:38.797974 898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:38.798278 898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:38.800711 898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d7b2088478e14951ba1d9acc7758f6d7"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-06l8"
I20240502 01:52:38.801424 898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d7b2088478e14951ba1d9acc7758f6d7"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-06l8"
I20240502 01:52:38.806223 898 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 01:52:38.809388 912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:38.810587 898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:52:38.810730 898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d7b2088478e14951ba1d9acc7758f6d7"
format_stamp: "Formatted at 2024-05-02 01:52:38 on dist-test-slave-06l8"
I20240502 01:52:38.810905 898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:38.842197 898 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:38.843070 898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:38.843340 898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:38.844250 898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:38.845727 898 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:38.845820 898 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:38.845930 898 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:38.845996 898 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:38.874871 898 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.193:44275
I20240502 01:52:38.874935 1024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.193:44275 every 8 connection(s)
I20240502 01:52:38.876127 898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:52:38.878319 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 898
I20240502 01:52:38.878506 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:52:38.882166 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.194:0
--local_ip_for_outbound_sockets=127.31.161.194
--webserver_interface=127.31.161.194
--webserver_port=0
--tserver_master_addrs=127.31.161.254:42383
--builtin_ntp_servers=127.31.161.212:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:38.889572 1025 heartbeater.cc:347] Connected to a master server at 127.31.161.254:42383
I20240502 01:52:38.889897 1025 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:38.890874 1025 heartbeater.cc:511] Master 127.31.161.254:42383 requested a full tablet report, sending...
I20240502 01:52:38.892949 839 ts_manager.cc:194] Registered new tserver with Master: d7b2088478e14951ba1d9acc7758f6d7 (127.31.161.193:44275)
I20240502 01:52:38.894387 839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.193:57231
W20240502 01:52:39.317807 1029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:39.318121 1029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:39.318256 1029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:39.325834 1029 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:39.325963 1029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:39.326102 1029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.194
I20240502 01:52:39.335757 1029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.31.161.194
--webserver_port=0
--tserver_master_addrs=127.31.161.254:42383
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.194
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:39.337201 1029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:39.339056 1029 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:39.351723 1035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:39.351815 1034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:39.352334 1037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:39.352703 1029 server_base.cc:1034] running on GCE node
I20240502 01:52:39.353329 1029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:39.354350 1029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:39.355563 1029 hybrid_clock.cc:648] HybridClock initialized: now 1714614759355517 us; error 36 us; skew 500 ppm
I20240502 01:52:39.355902 1029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:39.358284 1029 webserver.cc:460] Webserver started at http://127.31.161.194:41379/ using document root <none> and password file <none>
I20240502 01:52:39.359067 1029 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:39.359184 1029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:39.359508 1029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:39.362327 1029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "bf3e1eb9ed90430fbfd9e0fe33e7ec20"
format_stamp: "Formatted at 2024-05-02 01:52:39 on dist-test-slave-06l8"
I20240502 01:52:39.363085 1029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "bf3e1eb9ed90430fbfd9e0fe33e7ec20"
format_stamp: "Formatted at 2024-05-02 01:52:39 on dist-test-slave-06l8"
I20240502 01:52:39.368372 1029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 01:52:39.372031 1043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:39.373673 1029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:52:39.373837 1029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "bf3e1eb9ed90430fbfd9e0fe33e7ec20"
format_stamp: "Formatted at 2024-05-02 01:52:39 on dist-test-slave-06l8"
I20240502 01:52:39.374047 1029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:39.406237 1029 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:39.407126 1029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:39.407399 1029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:39.408442 1029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:39.410162 1029 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:39.410266 1029 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:39.410364 1029 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:39.410423 1029 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:39.440537 1029 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.194:37873
I20240502 01:52:39.440603 1157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.194:37873 every 8 connection(s)
I20240502 01:52:39.441890 1029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:52:39.442524 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 1029
I20240502 01:52:39.442688 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:52:39.446431 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.195:0
--local_ip_for_outbound_sockets=127.31.161.195
--webserver_interface=127.31.161.195
--webserver_port=0
--tserver_master_addrs=127.31.161.254:42383
--builtin_ntp_servers=127.31.161.212:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:39.458341 1158 heartbeater.cc:347] Connected to a master server at 127.31.161.254:42383
I20240502 01:52:39.458655 1158 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:39.459481 1158 heartbeater.cc:511] Master 127.31.161.254:42383 requested a full tablet report, sending...
I20240502 01:52:39.460947 839 ts_manager.cc:194] Registered new tserver with Master: bf3e1eb9ed90430fbfd9e0fe33e7ec20 (127.31.161.194:37873)
I20240502 01:52:39.462096 839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.194:48775
W20240502 01:52:39.775018 1162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:39.775388 1162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:39.775496 1162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:39.783314 1162 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:39.783445 1162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:39.783560 1162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.195
I20240502 01:52:39.793241 1162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.31.161.195
--webserver_port=0
--tserver_master_addrs=127.31.161.254:42383
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.195
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:39.794616 1162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:39.796449 1162 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:39.808568 1168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:39.809002 1170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:39.809194 1167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:39.809688 1162 server_base.cc:1034] running on GCE node
I20240502 01:52:39.810880 1162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:39.811859 1162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:39.813088 1162 hybrid_clock.cc:648] HybridClock initialized: now 1714614759813000 us; error 49 us; skew 500 ppm
I20240502 01:52:39.813511 1162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:39.815943 1162 webserver.cc:460] Webserver started at http://127.31.161.195:40967/ using document root <none> and password file <none>
I20240502 01:52:39.816720 1162 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:39.816834 1162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:39.817212 1162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:39.819998 1162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9e13160c306141ce9af88f2ca6cb2bc7"
format_stamp: "Formatted at 2024-05-02 01:52:39 on dist-test-slave-06l8"
I20240502 01:52:39.820771 1162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9e13160c306141ce9af88f2ca6cb2bc7"
format_stamp: "Formatted at 2024-05-02 01:52:39 on dist-test-slave-06l8"
I20240502 01:52:39.826313 1162 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 01:52:39.829854 1176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:39.831300 1162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:52:39.831473 1162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9e13160c306141ce9af88f2ca6cb2bc7"
format_stamp: "Formatted at 2024-05-02 01:52:39 on dist-test-slave-06l8"
I20240502 01:52:39.831670 1162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:39.849520 1162 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:39.850508 1162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:39.850800 1162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:39.851815 1162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:39.853440 1162 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:39.853543 1162 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:39.853657 1162 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:39.853732 1162 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:39.885857 1162 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.195:42007
I20240502 01:52:39.885911 1290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.195:42007 every 8 connection(s)
I20240502 01:52:39.887147 1162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:52:39.887531 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 1162
I20240502 01:52:39.887712 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:52:39.891153 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.196:0
--local_ip_for_outbound_sockets=127.31.161.196
--webserver_interface=127.31.161.196
--webserver_port=0
--tserver_master_addrs=127.31.161.254:42383
--builtin_ntp_servers=127.31.161.212:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:39.897696 1025 heartbeater.cc:503] Master 127.31.161.254:42383 was elected leader, sending a full tablet report...
I20240502 01:52:39.901832 1291 heartbeater.cc:347] Connected to a master server at 127.31.161.254:42383
I20240502 01:52:39.902135 1291 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:39.903064 1291 heartbeater.cc:511] Master 127.31.161.254:42383 requested a full tablet report, sending...
I20240502 01:52:39.904633 839 ts_manager.cc:194] Registered new tserver with Master: 9e13160c306141ce9af88f2ca6cb2bc7 (127.31.161.195:42007)
I20240502 01:52:39.905844 839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.195:60387
W20240502 01:52:40.227154 1294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:40.227495 1294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:40.227614 1294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:40.235421 1294 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:40.235579 1294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:40.235718 1294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.196
I20240502 01:52:40.245440 1294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.31.161.196
--webserver_port=0
--tserver_master_addrs=127.31.161.254:42383
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.196
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:40.246783 1294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:40.248692 1294 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:40.261775 1302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:40.262251 1299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:40.262558 1300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:40.263051 1294 server_base.cc:1034] running on GCE node
I20240502 01:52:40.263706 1294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:40.264698 1294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:40.265954 1294 hybrid_clock.cc:648] HybridClock initialized: now 1714614760265889 us; error 48 us; skew 500 ppm
I20240502 01:52:40.266309 1294 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:40.274564 1294 webserver.cc:460] Webserver started at http://127.31.161.196:38207/ using document root <none> and password file <none>
I20240502 01:52:40.275638 1294 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:40.275768 1294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:40.276163 1294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:40.279901 1294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "842812430adb4c2aa2645d9678f79f37"
format_stamp: "Formatted at 2024-05-02 01:52:40 on dist-test-slave-06l8"
I20240502 01:52:40.280776 1294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "842812430adb4c2aa2645d9678f79f37"
format_stamp: "Formatted at 2024-05-02 01:52:40 on dist-test-slave-06l8"
I20240502 01:52:40.286199 1294 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 01:52:40.289659 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:40.291378 1294 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:52:40.291541 1294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "842812430adb4c2aa2645d9678f79f37"
format_stamp: "Formatted at 2024-05-02 01:52:40 on dist-test-slave-06l8"
I20240502 01:52:40.291734 1294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:40.309895 1294 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:40.310791 1294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:40.311053 1294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:40.312072 1294 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:40.313692 1294 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:40.313782 1294 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:40.313899 1294 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:40.313971 1294 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:40.346988 1294 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.196:43995
I20240502 01:52:40.347039 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.196:43995 every 8 connection(s)
I20240502 01:52:40.348501 1294 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:52:40.351922 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 1294
I20240502 01:52:40.352074 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:52:40.355206 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.197:0
--local_ip_for_outbound_sockets=127.31.161.197
--webserver_interface=127.31.161.197
--webserver_port=0
--tserver_master_addrs=127.31.161.254:42383
--builtin_ntp_servers=127.31.161.212:36519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:40.363157 1422 heartbeater.cc:347] Connected to a master server at 127.31.161.254:42383
I20240502 01:52:40.363492 1422 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:40.364385 1422 heartbeater.cc:511] Master 127.31.161.254:42383 requested a full tablet report, sending...
I20240502 01:52:40.365859 839 ts_manager.cc:194] Registered new tserver with Master: 842812430adb4c2aa2645d9678f79f37 (127.31.161.196:43995)
I20240502 01:52:40.366711 839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.196:52373
I20240502 01:52:40.464624 1158 heartbeater.cc:503] Master 127.31.161.254:42383 was elected leader, sending a full tablet report...
W20240502 01:52:40.700384 1425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:40.700706 1425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:40.700842 1425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:40.708300 1425 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:40.708446 1425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:40.708596 1425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.197
I20240502 01:52:40.718689 1425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:36519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.31.161.197
--webserver_port=0
--tserver_master_addrs=127.31.161.254:42383
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.197
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:40.720126 1425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:40.721974 1425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:40.734005 1430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:40.734232 1433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:40.734632 1431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:40.735083 1425 server_base.cc:1034] running on GCE node
I20240502 01:52:40.735777 1425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:40.736814 1425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:40.738031 1425 hybrid_clock.cc:648] HybridClock initialized: now 1714614760737964 us; error 52 us; skew 500 ppm
I20240502 01:52:40.738341 1425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:40.740618 1425 webserver.cc:460] Webserver started at http://127.31.161.197:32889/ using document root <none> and password file <none>
I20240502 01:52:40.741465 1425 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:40.741586 1425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:40.741911 1425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:40.744597 1425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "15c60df199d64a6dbfcca074238aa050"
format_stamp: "Formatted at 2024-05-02 01:52:40 on dist-test-slave-06l8"
I20240502 01:52:40.745394 1425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "15c60df199d64a6dbfcca074238aa050"
format_stamp: "Formatted at 2024-05-02 01:52:40 on dist-test-slave-06l8"
I20240502 01:52:40.750485 1425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 01:52:40.753863 1439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:40.755174 1425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:52:40.755319 1425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "15c60df199d64a6dbfcca074238aa050"
format_stamp: "Formatted at 2024-05-02 01:52:40 on dist-test-slave-06l8"
I20240502 01:52:40.755518 1425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:40.784402 1425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:40.785363 1425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:40.785636 1425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:40.786589 1425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:40.788087 1425 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:40.788184 1425 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:40.788295 1425 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:40.788362 1425 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:40.818496 1425 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.197:36155
I20240502 01:52:40.818564 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.197:36155 every 8 connection(s)
I20240502 01:52:40.819828 1425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:52:40.825433 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 1425
I20240502 01:52:40.825634 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:52:40.833806 1553 heartbeater.cc:347] Connected to a master server at 127.31.161.254:42383
I20240502 01:52:40.834107 1553 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:40.834923 1553 heartbeater.cc:511] Master 127.31.161.254:42383 requested a full tablet report, sending...
I20240502 01:52:40.836268 839 ts_manager.cc:194] Registered new tserver with Master: 15c60df199d64a6dbfcca074238aa050 (127.31.161.197:36155)
I20240502 01:52:40.837124 839 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.197:53435
I20240502 01:52:40.841536 32391 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 01:52:40.856701 839 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49534:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 01:52:40.894718 1091 tablet_service.cc:1467] Processing CreateTablet for tablet f6d87832b448471b80d0bc5c42620d00 (DEFAULT_TABLE table=TestTable [id=dd581882e6274c928bceffc64643f6b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:40.896358 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d87832b448471b80d0bc5c42620d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:40.896749 1357 tablet_service.cc:1467] Processing CreateTablet for tablet f6d87832b448471b80d0bc5c42620d00 (DEFAULT_TABLE table=TestTable [id=dd581882e6274c928bceffc64643f6b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:40.898283 1357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d87832b448471b80d0bc5c42620d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:40.900296 1488 tablet_service.cc:1467] Processing CreateTablet for tablet f6d87832b448471b80d0bc5c42620d00 (DEFAULT_TABLE table=TestTable [id=dd581882e6274c928bceffc64643f6b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:40.901620 1488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6d87832b448471b80d0bc5c42620d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:40.908556 1291 heartbeater.cc:503] Master 127.31.161.254:42383 was elected leader, sending a full tablet report...
I20240502 01:52:40.912614 1572 tablet_bootstrap.cc:492] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050: Bootstrap starting.
I20240502 01:52:40.915963 1571 tablet_bootstrap.cc:492] T f6d87832b448471b80d0bc5c42620d00 P bf3e1eb9ed90430fbfd9e0fe33e7ec20: Bootstrap starting.
I20240502 01:52:40.916741 1572 tablet_bootstrap.cc:654] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:40.918602 1573 tablet_bootstrap.cc:492] T f6d87832b448471b80d0bc5c42620d00 P 842812430adb4c2aa2645d9678f79f37: Bootstrap starting.
I20240502 01:52:40.919435 1572 log.cc:824] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:40.919520 1571 tablet_bootstrap.cc:654] T f6d87832b448471b80d0bc5c42620d00 P bf3e1eb9ed90430fbfd9e0fe33e7ec20: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:40.920687 1571 log.cc:824] T f6d87832b448471b80d0bc5c42620d00 P bf3e1eb9ed90430fbfd9e0fe33e7ec20: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:40.921649 1573 tablet_bootstrap.cc:654] T f6d87832b448471b80d0bc5c42620d00 P 842812430adb4c2aa2645d9678f79f37: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:40.922689 1572 tablet_bootstrap.cc:492] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050: No bootstrap required, opened a new log
I20240502 01:52:40.922729 1573 log.cc:824] T f6d87832b448471b80d0bc5c42620d00 P 842812430adb4c2aa2645d9678f79f37: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:40.922971 1572 ts_tablet_manager.cc:1367] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20240502 01:52:40.924122 1571 tablet_bootstrap.cc:492] T f6d87832b448471b80d0bc5c42620d00 P bf3e1eb9ed90430fbfd9e0fe33e7ec20: No bootstrap required, opened a new log
I20240502 01:52:40.924394 1571 ts_tablet_manager.cc:1367] T f6d87832b448471b80d0bc5c42620d00 P bf3e1eb9ed90430fbfd9e0fe33e7ec20: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20240502 01:52:40.924996 1573 tablet_bootstrap.cc:492] T f6d87832b448471b80d0bc5c42620d00 P 842812430adb4c2aa2645d9678f79f37: No bootstrap required, opened a new log
I20240502 01:52:40.925276 1573 ts_tablet_manager.cc:1367] T f6d87832b448471b80d0bc5c42620d00 P 842812430adb4c2aa2645d9678f79f37: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.005s
I20240502 01:52:40.931808 1573 raft_consensus.cc:348] T f6d87832b448471b80d0bc5c42620d00 P 842812430adb4c2aa2645d9678f79f37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3e1eb9ed90430fbfd9e0fe33e7ec20" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 37873 } } peers { permanent_uuid: "842812430adb4c2aa2645d9678f79f37" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 43995 } } peers { permanent_uuid: "15c60df199d64a6dbfcca074238aa050" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 36155 } }
I20240502 01:52:40.932173 1573 raft_consensus.cc:374] T f6d87832b448471b80d0bc5c42620d00 P 842812430adb4c2aa2645d9678f79f37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:40.932327 1573 raft_consensus.cc:729] T f6d87832b448471b80d0bc5c42620d00 P 842812430adb4c2aa2645d9678f79f37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842812430adb4c2aa2645d9678f79f37, State: Initialized, Role: FOLLOWER
I20240502 01:52:40.932385 1572 raft_consensus.cc:348] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3e1eb9ed90430fbfd9e0fe33e7ec20" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 37873 } } peers { permanent_uuid: "842812430adb4c2aa2645d9678f79f37" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 43995 } } peers { permanent_uuid: "15c60df199d64a6dbfcca074238aa050" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 36155 } }
I20240502 01:52:40.932849 1572 raft_consensus.cc:374] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:40.932958 1572 raft_consensus.cc:729] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15c60df199d64a6dbfcca074238aa050, State: Initialized, Role: FOLLOWER
I20240502 01:52:40.933010 1573 consensus_queue.cc:260] T f6d87832b448471b80d0bc5c42620d00 P 842812430adb4c2aa2645d9678f79f37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3e1eb9ed90430fbfd9e0fe33e7ec20" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 37873 } } peers { permanent_uuid: "842812430adb4c2aa2645d9678f79f37" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 43995 } } peers { permanent_uuid: "15c60df199d64a6dbfcca074238aa050" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 36155 } }
I20240502 01:52:40.932860 1571 raft_consensus.cc:348] T f6d87832b448471b80d0bc5c42620d00 P bf3e1eb9ed90430fbfd9e0fe33e7ec20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3e1eb9ed90430fbfd9e0fe33e7ec20" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 37873 } } peers { permanent_uuid: "842812430adb4c2aa2645d9678f79f37" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 43995 } } peers { permanent_uuid: "15c60df199d64a6dbfcca074238aa050" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 36155 } }
I20240502 01:52:40.933295 1571 raft_consensus.cc:374] T f6d87832b448471b80d0bc5c42620d00 P bf3e1eb9ed90430fbfd9e0fe33e7ec20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:40.933404 1571 raft_consensus.cc:729] T f6d87832b448471b80d0bc5c42620d00 P bf3e1eb9ed90430fbfd9e0fe33e7ec20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf3e1eb9ed90430fbfd9e0fe33e7ec20, State: Initialized, Role: FOLLOWER
I20240502 01:52:40.933779 1572 consensus_queue.cc:260] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3e1eb9ed90430fbfd9e0fe33e7ec20" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 37873 } } peers { permanent_uuid: "842812430adb4c2aa2645d9678f79f37" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 43995 } } peers { permanent_uuid: "15c60df199d64a6dbfcca074238aa050" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 36155 } }
I20240502 01:52:40.934046 1571 consensus_queue.cc:260] T f6d87832b448471b80d0bc5c42620d00 P bf3e1eb9ed90430fbfd9e0fe33e7ec20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3e1eb9ed90430fbfd9e0fe33e7ec20" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 37873 } } peers { permanent_uuid: "842812430adb4c2aa2645d9678f79f37" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 43995 } } peers { permanent_uuid: "15c60df199d64a6dbfcca074238aa050" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 36155 } }
I20240502 01:52:40.938037 1553 heartbeater.cc:503] Master 127.31.161.254:42383 was elected leader, sending a full tablet report...
I20240502 01:52:40.939157 1572 ts_tablet_manager.cc:1398] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050: Time spent starting tablet: real 0.016s user 0.004s sys 0.012s
I20240502 01:52:40.940235 1422 heartbeater.cc:503] Master 127.31.161.254:42383 was elected leader, sending a full tablet report...
I20240502 01:52:40.940840 1573 ts_tablet_manager.cc:1398] T f6d87832b448471b80d0bc5c42620d00 P 842812430adb4c2aa2645d9678f79f37: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20240502 01:52:40.942420 1571 ts_tablet_manager.cc:1398] T f6d87832b448471b80d0bc5c42620d00 P bf3e1eb9ed90430fbfd9e0fe33e7ec20: Time spent starting tablet: real 0.018s user 0.008s sys 0.011s
W20240502 01:52:40.944828 1159 tablet.cc:2369] T f6d87832b448471b80d0bc5c42620d00 P bf3e1eb9ed90430fbfd9e0fe33e7ec20: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:52:41.073810 1554 tablet.cc:2369] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:52:41.100562 1423 tablet.cc:2369] T f6d87832b448471b80d0bc5c42620d00 P 842812430adb4c2aa2645d9678f79f37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:52:41.128305 1578 raft_consensus.cc:482] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:41.128564 1578 raft_consensus.cc:504] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3e1eb9ed90430fbfd9e0fe33e7ec20" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 37873 } } peers { permanent_uuid: "842812430adb4c2aa2645d9678f79f37" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 43995 } } peers { permanent_uuid: "15c60df199d64a6dbfcca074238aa050" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 36155 } }
I20240502 01:52:41.130368 1578 leader_election.cc:288] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf3e1eb9ed90430fbfd9e0fe33e7ec20 (127.31.161.194:37873), 842812430adb4c2aa2645d9678f79f37 (127.31.161.196:43995)
I20240502 01:52:41.137425 1112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d87832b448471b80d0bc5c42620d00" candidate_uuid: "15c60df199d64a6dbfcca074238aa050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf3e1eb9ed90430fbfd9e0fe33e7ec20" is_pre_election: true
I20240502 01:52:41.137990 1112 raft_consensus.cc:2454] T f6d87832b448471b80d0bc5c42620d00 P bf3e1eb9ed90430fbfd9e0fe33e7ec20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15c60df199d64a6dbfcca074238aa050 in term 0.
I20240502 01:52:41.138103 1377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d87832b448471b80d0bc5c42620d00" candidate_uuid: "15c60df199d64a6dbfcca074238aa050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842812430adb4c2aa2645d9678f79f37" is_pre_election: true
I20240502 01:52:41.138598 1377 raft_consensus.cc:2454] T f6d87832b448471b80d0bc5c42620d00 P 842812430adb4c2aa2645d9678f79f37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15c60df199d64a6dbfcca074238aa050 in term 0.
I20240502 01:52:41.138779 1442 leader_election.cc:302] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15c60df199d64a6dbfcca074238aa050, bf3e1eb9ed90430fbfd9e0fe33e7ec20; no voters:
I20240502 01:52:41.139191 1578 raft_consensus.cc:2781] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:41.139344 1578 raft_consensus.cc:482] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:41.139456 1578 raft_consensus.cc:3037] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:41.141511 1578 raft_consensus.cc:504] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3e1eb9ed90430fbfd9e0fe33e7ec20" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 37873 } } peers { permanent_uuid: "842812430adb4c2aa2645d9678f79f37" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 43995 } } peers { permanent_uuid: "15c60df199d64a6dbfcca074238aa050" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 36155 } }
I20240502 01:52:41.142131 1578 leader_election.cc:288] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050 [CANDIDATE]: Term 1 election: Requested vote from peers bf3e1eb9ed90430fbfd9e0fe33e7ec20 (127.31.161.194:37873), 842812430adb4c2aa2645d9678f79f37 (127.31.161.196:43995)
I20240502 01:52:41.142493 1112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d87832b448471b80d0bc5c42620d00" candidate_uuid: "15c60df199d64a6dbfcca074238aa050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf3e1eb9ed90430fbfd9e0fe33e7ec20"
I20240502 01:52:41.142591 1377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6d87832b448471b80d0bc5c42620d00" candidate_uuid: "15c60df199d64a6dbfcca074238aa050" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "842812430adb4c2aa2645d9678f79f37"
I20240502 01:52:41.142704 1112 raft_consensus.cc:3037] T f6d87832b448471b80d0bc5c42620d00 P bf3e1eb9ed90430fbfd9e0fe33e7ec20 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:41.142778 1377 raft_consensus.cc:3037] T f6d87832b448471b80d0bc5c42620d00 P 842812430adb4c2aa2645d9678f79f37 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:41.144757 1112 raft_consensus.cc:2454] T f6d87832b448471b80d0bc5c42620d00 P bf3e1eb9ed90430fbfd9e0fe33e7ec20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15c60df199d64a6dbfcca074238aa050 in term 1.
I20240502 01:52:41.144845 1377 raft_consensus.cc:2454] T f6d87832b448471b80d0bc5c42620d00 P 842812430adb4c2aa2645d9678f79f37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15c60df199d64a6dbfcca074238aa050 in term 1.
I20240502 01:52:41.145124 1442 leader_election.cc:302] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15c60df199d64a6dbfcca074238aa050, bf3e1eb9ed90430fbfd9e0fe33e7ec20; no voters:
I20240502 01:52:41.145412 1578 raft_consensus.cc:2781] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:41.145843 1578 raft_consensus.cc:686] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050 [term 1 LEADER]: Becoming Leader. State: Replica: 15c60df199d64a6dbfcca074238aa050, State: Running, Role: LEADER
I20240502 01:52:41.146318 1578 consensus_queue.cc:237] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3e1eb9ed90430fbfd9e0fe33e7ec20" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 37873 } } peers { permanent_uuid: "842812430adb4c2aa2645d9678f79f37" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 43995 } } peers { permanent_uuid: "15c60df199d64a6dbfcca074238aa050" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 36155 } }
I20240502 01:52:41.150761 837 catalog_manager.cc:5484] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15c60df199d64a6dbfcca074238aa050 (127.31.161.197). New cstate: current_term: 1 leader_uuid: "15c60df199d64a6dbfcca074238aa050" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf3e1eb9ed90430fbfd9e0fe33e7ec20" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 37873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "842812430adb4c2aa2645d9678f79f37" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 43995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15c60df199d64a6dbfcca074238aa050" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 36155 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:41.185699 32391 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 01:52:41.188167 32391 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d7b2088478e14951ba1d9acc7758f6d7 to finish bootstrapping
I20240502 01:52:41.197716 32391 ts_itest-base.cc:233] Waiting for 1 tablets on tserver bf3e1eb9ed90430fbfd9e0fe33e7ec20 to finish bootstrapping
I20240502 01:52:41.205436 32391 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 9e13160c306141ce9af88f2ca6cb2bc7 to finish bootstrapping
I20240502 01:52:41.213776 32391 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 842812430adb4c2aa2645d9678f79f37 to finish bootstrapping
I20240502 01:52:41.220918 32391 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 15c60df199d64a6dbfcca074238aa050 to finish bootstrapping
I20240502 01:52:41.682675 1226 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:41.684803 1488 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:41.687322 1091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:41.690230 1619 consensus_queue.cc:1035] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050 [LEADER]: Connected to new peer: Peer: permanent_uuid: "842812430adb4c2aa2645d9678f79f37" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:41.690439 1357 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:41.692241 960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:41.744659 1625 consensus_queue.cc:1035] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf3e1eb9ed90430fbfd9e0fe33e7ec20" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:41.866077 1488 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:41.870867 1357 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:41.881973 1226 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:41.882898 1091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:41.896078 960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:42.053229 1488 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:42.064301 1226 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:42.065318 960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:42.075109 1357 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:42.075539 1091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:42.649032 1488 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:42.658762 1226 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:42.660200 1357 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:42.660390 1091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:42.669461 960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:42.818292 1357 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:42.820606 1226 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:42.822014 1488 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:42.837685 1091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:42.838289 960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:43.001163 1488 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:52:43.010000 1357 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:43.011468 1091 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:43.013831 1226 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:43.018477 960 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:52:43.166075 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 898
I20240502 01:52:43.167501 1019 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:43.256606 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 898
I20240502 01:52:43.272199 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 1029
I20240502 01:52:43.273213 1152 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:43.377226 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 1029
I20240502 01:52:43.392719 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 1162
I20240502 01:52:43.393757 1286 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:52:43.401044 1442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.161.194:37873: connect: Connection refused (error 111)
W20240502 01:52:43.402595 1442 consensus_peers.cc:485] T f6d87832b448471b80d0bc5c42620d00 P 15c60df199d64a6dbfcca074238aa050 -> Peer bf3e1eb9ed90430fbfd9e0fe33e7ec20 (127.31.161.194:37873): Couldn't send request to peer bf3e1eb9ed90430fbfd9e0fe33e7ec20. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.194:37873: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:52:43.487073 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 1162
I20240502 01:52:43.502485 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 1294
I20240502 01:52:43.503507 1417 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:43.599117 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 1294
I20240502 01:52:43.614195 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 1425
I20240502 01:52:43.615181 1548 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:43.709544 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 1425
I20240502 01:52:43.725030 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 808
I20240502 01:52:43.726260 869 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:43.838881 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 808
2024-05-02T01:52:43Z chronyd exiting
[ OK ] RebalanceStartSafetyTest.TooManyIgnoredTservers/1 (6040 ms)
[----------] 1 test from RebalanceStartSafetyTest (6040 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/7
I20240502 01:52:43.878661 32391 test_util.cc:268] Using random seed: -605194013
I20240502 01:52:43.880411 32391 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:52:43.880487 32391 ts_itest-base.cc:113] --------------
I20240502 01:52:43.880532 32391 ts_itest-base.cc:114] 7 tablet servers
I20240502 01:52:43.880595 32391 ts_itest-base.cc:115] 5 replicas per TS
I20240502 01:52:43.880645 32391 ts_itest-base.cc:116] --------------
2024-05-02T01:52:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:52:43Z Disabled control of system clock
I20240502 01:52:43.901887 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.161.254:45525
--webserver_interface=127.31.161.254
--webserver_port=0
--builtin_ntp_servers=127.31.161.212:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.161.254:45525
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 01:52:44.208530 1819 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 01:52:44.208833 1819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:44.208946 1819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:44.209044 1819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:44.216425 1819 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:44.216539 1819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:52:44.216632 1819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:44.216691 1819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:52:44.216771 1819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:52:44.225656 1819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.31.161.254:45525
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.161.254:45525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.31.161.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:44.226977 1819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:44.228657 1819 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:44.239137 1825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:44.239714 1824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:44.240027 1827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:44.240434 1819 server_base.cc:1034] running on GCE node
I20240502 01:52:44.241181 1819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:44.242444 1819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:44.243655 1819 hybrid_clock.cc:648] HybridClock initialized: now 1714614764243592 us; error 40 us; skew 500 ppm
I20240502 01:52:44.243944 1819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:44.245987 1819 webserver.cc:460] Webserver started at http://127.31.161.254:34619/ using document root <none> and password file <none>
I20240502 01:52:44.246650 1819 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:44.246774 1819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:44.247047 1819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:44.249541 1819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4689208abb42472a9ef49d5db3ba5cda"
format_stamp: "Formatted at 2024-05-02 01:52:44 on dist-test-slave-06l8"
I20240502 01:52:44.250236 1819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4689208abb42472a9ef49d5db3ba5cda"
format_stamp: "Formatted at 2024-05-02 01:52:44 on dist-test-slave-06l8"
I20240502 01:52:44.254999 1819 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 01:52:44.258152 1833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:44.259374 1819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:52:44.259541 1819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4689208abb42472a9ef49d5db3ba5cda"
format_stamp: "Formatted at 2024-05-02 01:52:44 on dist-test-slave-06l8"
I20240502 01:52:44.259706 1819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:44.296617 1819 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:44.297415 1819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:44.297693 1819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:44.314162 1819 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.254:45525
I20240502 01:52:44.314198 1884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.254:45525 every 8 connection(s)
I20240502 01:52:44.315390 1819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:52:44.315725 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 1819
I20240502 01:52:44.315922 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:52:44.319494 1885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:44.329978 1885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda: Bootstrap starting.
I20240502 01:52:44.332623 1885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:44.333586 1885 log.cc:824] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:44.335992 1885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda: No bootstrap required, opened a new log
I20240502 01:52:44.341537 1885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4689208abb42472a9ef49d5db3ba5cda" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 45525 } }
I20240502 01:52:44.341889 1885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:44.342016 1885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4689208abb42472a9ef49d5db3ba5cda, State: Initialized, Role: FOLLOWER
I20240502 01:52:44.342543 1885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4689208abb42472a9ef49d5db3ba5cda" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 45525 } }
I20240502 01:52:44.342732 1885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:52:44.342844 1885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:52:44.342993 1885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:44.344839 1885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4689208abb42472a9ef49d5db3ba5cda" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 45525 } }
I20240502 01:52:44.345356 1885 leader_election.cc:302] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4689208abb42472a9ef49d5db3ba5cda; no voters:
I20240502 01:52:44.345856 1885 leader_election.cc:288] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:52:44.345980 1890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:44.346534 1890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda [term 1 LEADER]: Becoming Leader. State: Replica: 4689208abb42472a9ef49d5db3ba5cda, State: Running, Role: LEADER
I20240502 01:52:44.347054 1890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4689208abb42472a9ef49d5db3ba5cda" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 45525 } }
I20240502 01:52:44.347674 1885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:52:44.349834 1892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4689208abb42472a9ef49d5db3ba5cda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4689208abb42472a9ef49d5db3ba5cda" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 45525 } } }
I20240502 01:52:44.350189 1892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:44.350823 1893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4689208abb42472a9ef49d5db3ba5cda. Latest consensus state: current_term: 1 leader_uuid: "4689208abb42472a9ef49d5db3ba5cda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4689208abb42472a9ef49d5db3ba5cda" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 45525 } } }
I20240502 01:52:44.351159 1893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda [sys.catalog]: This master's current role is: LEADER
I20240502 01:52:44.351418 1899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:52:44.356401 1899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:52:44.364480 1899 catalog_manager.cc:1347] Generated new cluster ID: 2af0de813d6f4db7990367649196879c
I20240502 01:52:44.364651 1899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:52:44.375772 1899 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:52:44.376861 1899 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:52:44.384567 1899 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4689208abb42472a9ef49d5db3ba5cda: Generated new TSK 0
I20240502 01:52:44.385285 1899 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:52:44.395363 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.193:0
--local_ip_for_outbound_sockets=127.31.161.193
--webserver_interface=127.31.161.193
--webserver_port=0
--tserver_master_addrs=127.31.161.254:45525
--builtin_ntp_servers=127.31.161.212:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:52:44.743664 1910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:44.743943 1910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:44.744067 1910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:44.751368 1910 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:44.751494 1910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:44.751653 1910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.193
I20240502 01:52:44.760249 1910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.31.161.193
--webserver_port=0
--tserver_master_addrs=127.31.161.254:45525
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.193
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:44.761483 1910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:44.763105 1910 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:44.774606 1915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:44.774976 1918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:44.775314 1916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:44.776003 1910 server_base.cc:1034] running on GCE node
I20240502 01:52:44.776520 1910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:44.777340 1910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:44.778549 1910 hybrid_clock.cc:648] HybridClock initialized: now 1714614764778490 us; error 42 us; skew 500 ppm
I20240502 01:52:44.778952 1910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:44.780965 1910 webserver.cc:460] Webserver started at http://127.31.161.193:33577/ using document root <none> and password file <none>
I20240502 01:52:44.781641 1910 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:44.781730 1910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:44.782006 1910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:44.784386 1910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "58dae5dc94444d5e92e6907c5b637879"
format_stamp: "Formatted at 2024-05-02 01:52:44 on dist-test-slave-06l8"
I20240502 01:52:44.785037 1910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "58dae5dc94444d5e92e6907c5b637879"
format_stamp: "Formatted at 2024-05-02 01:52:44 on dist-test-slave-06l8"
I20240502 01:52:44.789599 1910 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 01:52:44.792608 1924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:44.793830 1910 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:52:44.793969 1910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "58dae5dc94444d5e92e6907c5b637879"
format_stamp: "Formatted at 2024-05-02 01:52:44 on dist-test-slave-06l8"
I20240502 01:52:44.794139 1910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:44.808344 1910 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:44.809041 1910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:44.809314 1910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:44.810142 1910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:44.811470 1910 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:44.811553 1910 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:44.811654 1910 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:44.811729 1910 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:44.839740 1910 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.193:39307
I20240502 01:52:44.839797 2036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.193:39307 every 8 connection(s)
I20240502 01:52:44.840947 1910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:52:44.844193 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 1910
I20240502 01:52:44.844413 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:52:44.848349 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.194:0
--local_ip_for_outbound_sockets=127.31.161.194
--webserver_interface=127.31.161.194
--webserver_port=0
--tserver_master_addrs=127.31.161.254:45525
--builtin_ntp_servers=127.31.161.212:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:44.855480 2037 heartbeater.cc:347] Connected to a master server at 127.31.161.254:45525
I20240502 01:52:44.855757 2037 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:44.856535 2037 heartbeater.cc:511] Master 127.31.161.254:45525 requested a full tablet report, sending...
I20240502 01:52:44.858762 1850 ts_manager.cc:194] Registered new tserver with Master: 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307)
I20240502 01:52:44.860776 1850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.193:34207
W20240502 01:52:45.165357 2041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:45.165627 2041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:45.165730 2041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:45.173192 2041 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:45.173306 2041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:45.173429 2041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.194
I20240502 01:52:45.182443 2041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.31.161.194
--webserver_port=0
--tserver_master_addrs=127.31.161.254:45525
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.194
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:45.183727 2041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:45.185415 2041 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:45.196607 2046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:45.197119 2049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:45.197512 2047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:45.198195 2041 server_base.cc:1034] running on GCE node
I20240502 01:52:45.198710 2041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:45.199599 2041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:45.200798 2041 hybrid_clock.cc:648] HybridClock initialized: now 1714614765200761 us; error 31 us; skew 500 ppm
I20240502 01:52:45.201133 2041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:45.203191 2041 webserver.cc:460] Webserver started at http://127.31.161.194:44673/ using document root <none> and password file <none>
I20240502 01:52:45.203859 2041 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:45.203963 2041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:45.204250 2041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:45.206687 2041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "269c9061e1a1435e92f79f0b9486dfea"
format_stamp: "Formatted at 2024-05-02 01:52:45 on dist-test-slave-06l8"
I20240502 01:52:45.207386 2041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "269c9061e1a1435e92f79f0b9486dfea"
format_stamp: "Formatted at 2024-05-02 01:52:45 on dist-test-slave-06l8"
I20240502 01:52:45.211942 2041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240502 01:52:45.215080 2055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:45.216276 2041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:45.216434 2041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "269c9061e1a1435e92f79f0b9486dfea"
format_stamp: "Formatted at 2024-05-02 01:52:45 on dist-test-slave-06l8"
I20240502 01:52:45.216610 2041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:45.231807 2041 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:45.232513 2041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:45.232759 2041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:45.233626 2041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:45.234990 2041 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:45.235086 2041 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:45.235185 2041 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:45.235256 2041 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:45.263906 2041 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.194:39621
I20240502 01:52:45.264001 2168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.194:39621 every 8 connection(s)
I20240502 01:52:45.265244 2041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:52:45.275359 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 2041
I20240502 01:52:45.275542 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:52:45.278508 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.195:0
--local_ip_for_outbound_sockets=127.31.161.195
--webserver_interface=127.31.161.195
--webserver_port=0
--tserver_master_addrs=127.31.161.254:45525
--builtin_ntp_servers=127.31.161.212:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:45.281780 2169 heartbeater.cc:347] Connected to a master server at 127.31.161.254:45525
I20240502 01:52:45.282096 2169 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:45.283025 2169 heartbeater.cc:511] Master 127.31.161.254:45525 requested a full tablet report, sending...
I20240502 01:52:45.284539 1850 ts_manager.cc:194] Registered new tserver with Master: 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621)
I20240502 01:52:45.285400 1850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.194:37709
W20240502 01:52:45.591647 2172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:45.591920 2172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:45.592048 2172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:45.599301 2172 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:45.599428 2172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:45.599581 2172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.195
I20240502 01:52:45.608062 2172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.31.161.195
--webserver_port=0
--tserver_master_addrs=127.31.161.254:45525
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.195
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:45.609303 2172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:45.610970 2172 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:45.622313 2177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:45.622802 2180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:45.623122 2178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:45.623601 2172 server_base.cc:1034] running on GCE node
I20240502 01:52:45.624130 2172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:45.624984 2172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:45.626185 2172 hybrid_clock.cc:648] HybridClock initialized: now 1714614765626136 us; error 44 us; skew 500 ppm
I20240502 01:52:45.626492 2172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:45.628597 2172 webserver.cc:460] Webserver started at http://127.31.161.195:40101/ using document root <none> and password file <none>
I20240502 01:52:45.629314 2172 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:45.629431 2172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:45.629717 2172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:45.632138 2172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "446c7f9bf31a4acc9dae8daf43330906"
format_stamp: "Formatted at 2024-05-02 01:52:45 on dist-test-slave-06l8"
I20240502 01:52:45.632848 2172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "446c7f9bf31a4acc9dae8daf43330906"
format_stamp: "Formatted at 2024-05-02 01:52:45 on dist-test-slave-06l8"
I20240502 01:52:45.637640 2172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 01:52:45.640767 2186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:45.642040 2172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:52:45.642181 2172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "446c7f9bf31a4acc9dae8daf43330906"
format_stamp: "Formatted at 2024-05-02 01:52:45 on dist-test-slave-06l8"
I20240502 01:52:45.642357 2172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:45.657971 2172 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:45.658833 2172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:45.659106 2172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:45.660076 2172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:45.661578 2172 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:45.661666 2172 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:45.661775 2172 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:45.661854 2172 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:45.690757 2172 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.195:39353
I20240502 01:52:45.690857 2300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.195:39353 every 8 connection(s)
I20240502 01:52:45.692015 2172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:52:45.695755 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 2172
I20240502 01:52:45.695940 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:52:45.699347 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.196:0
--local_ip_for_outbound_sockets=127.31.161.196
--webserver_interface=127.31.161.196
--webserver_port=0
--tserver_master_addrs=127.31.161.254:45525
--builtin_ntp_servers=127.31.161.212:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:45.705718 2301 heartbeater.cc:347] Connected to a master server at 127.31.161.254:45525
I20240502 01:52:45.705982 2301 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:45.706714 2301 heartbeater.cc:511] Master 127.31.161.254:45525 requested a full tablet report, sending...
I20240502 01:52:45.708043 1850 ts_manager.cc:194] Registered new tserver with Master: 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353)
I20240502 01:52:45.708801 1850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.195:58523
I20240502 01:52:45.863632 2037 heartbeater.cc:503] Master 127.31.161.254:45525 was elected leader, sending a full tablet report...
W20240502 01:52:46.007879 2304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:46.008167 2304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:46.008260 2304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:46.015456 2304 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:46.015574 2304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:46.015690 2304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.196
I20240502 01:52:46.024363 2304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.31.161.196
--webserver_port=0
--tserver_master_addrs=127.31.161.254:45525
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.196
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:46.025642 2304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:46.027298 2304 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:46.038758 2310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:46.039075 2309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:46.039383 2312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:46.039872 2304 server_base.cc:1034] running on GCE node
I20240502 01:52:46.040412 2304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:46.041309 2304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:46.042507 2304 hybrid_clock.cc:648] HybridClock initialized: now 1714614766042456 us; error 46 us; skew 500 ppm
I20240502 01:52:46.042806 2304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:46.044925 2304 webserver.cc:460] Webserver started at http://127.31.161.196:44429/ using document root <none> and password file <none>
I20240502 01:52:46.045684 2304 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:46.045796 2304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:46.046114 2304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:46.048751 2304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "7ef43ea75ecb46ee905b781c91aed95a"
format_stamp: "Formatted at 2024-05-02 01:52:46 on dist-test-slave-06l8"
I20240502 01:52:46.049592 2304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "7ef43ea75ecb46ee905b781c91aed95a"
format_stamp: "Formatted at 2024-05-02 01:52:46 on dist-test-slave-06l8"
I20240502 01:52:46.055094 2304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20240502 01:52:46.058604 2318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:46.060019 2304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 01:52:46.060186 2304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "7ef43ea75ecb46ee905b781c91aed95a"
format_stamp: "Formatted at 2024-05-02 01:52:46 on dist-test-slave-06l8"
I20240502 01:52:46.060381 2304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:46.101410 2304 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:46.102293 2304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:46.102562 2304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:46.103539 2304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:46.104993 2304 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:46.105114 2304 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:46.105227 2304 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:46.105301 2304 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:46.134423 2304 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.196:39225
I20240502 01:52:46.134492 2432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.196:39225 every 8 connection(s)
I20240502 01:52:46.135689 2304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:52:46.145864 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 2304
I20240502 01:52:46.146064 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:52:46.148526 2433 heartbeater.cc:347] Connected to a master server at 127.31.161.254:45525
I20240502 01:52:46.148857 2433 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:46.149473 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.197:0
--local_ip_for_outbound_sockets=127.31.161.197
--webserver_interface=127.31.161.197
--webserver_port=0
--tserver_master_addrs=127.31.161.254:45525
--builtin_ntp_servers=127.31.161.212:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:46.149799 2433 heartbeater.cc:511] Master 127.31.161.254:45525 requested a full tablet report, sending...
I20240502 01:52:46.151271 1850 ts_manager.cc:194] Registered new tserver with Master: 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225)
I20240502 01:52:46.152132 1850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.196:56985
I20240502 01:52:46.288149 2169 heartbeater.cc:503] Master 127.31.161.254:45525 was elected leader, sending a full tablet report...
W20240502 01:52:46.474049 2436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:46.474324 2436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:46.474455 2436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:46.482779 2436 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:46.482908 2436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:46.483064 2436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.197
I20240502 01:52:46.492301 2436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.31.161.197
--webserver_port=0
--tserver_master_addrs=127.31.161.254:45525
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.197
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:46.493649 2436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:46.495553 2436 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:46.581019 2441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:46.581387 2442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:46.581774 2444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:46.582247 2436 server_base.cc:1034] running on GCE node
I20240502 01:52:46.582886 2436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:46.583848 2436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:46.585052 2436 hybrid_clock.cc:648] HybridClock initialized: now 1714614766584990 us; error 56 us; skew 500 ppm
I20240502 01:52:46.585425 2436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:46.587662 2436 webserver.cc:460] Webserver started at http://127.31.161.197:39265/ using document root <none> and password file <none>
I20240502 01:52:46.588423 2436 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:46.588528 2436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:46.588816 2436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:46.591496 2436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "4d99e1fdda704211b3d101f892c57119"
format_stamp: "Formatted at 2024-05-02 01:52:46 on dist-test-slave-06l8"
I20240502 01:52:46.592237 2436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "4d99e1fdda704211b3d101f892c57119"
format_stamp: "Formatted at 2024-05-02 01:52:46 on dist-test-slave-06l8"
I20240502 01:52:46.598448 2436 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20240502 01:52:46.602226 2450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:46.603659 2436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:52:46.603823 2436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "4d99e1fdda704211b3d101f892c57119"
format_stamp: "Formatted at 2024-05-02 01:52:46 on dist-test-slave-06l8"
I20240502 01:52:46.604024 2436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:46.618862 2436 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:46.619635 2436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:46.619910 2436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:46.620831 2436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:46.622284 2436 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:46.622376 2436 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:46.622489 2436 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:46.622570 2436 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:46.652637 2436 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.197:38987
I20240502 01:52:46.652699 2563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.197:38987 every 8 connection(s)
I20240502 01:52:46.653988 2436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:52:46.656653 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 2436
I20240502 01:52:46.656829 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:52:46.660357 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.198:0
--local_ip_for_outbound_sockets=127.31.161.198
--webserver_interface=127.31.161.198
--webserver_port=0
--tserver_master_addrs=127.31.161.254:45525
--builtin_ntp_servers=127.31.161.212:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:46.667464 2564 heartbeater.cc:347] Connected to a master server at 127.31.161.254:45525
I20240502 01:52:46.667789 2564 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:46.668728 2564 heartbeater.cc:511] Master 127.31.161.254:45525 requested a full tablet report, sending...
I20240502 01:52:46.670257 1850 ts_manager.cc:194] Registered new tserver with Master: 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987)
I20240502 01:52:46.671087 1850 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.197:45195
I20240502 01:52:46.711136 2301 heartbeater.cc:503] Master 127.31.161.254:45525 was elected leader, sending a full tablet report...
W20240502 01:52:46.982894 2568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:46.983180 2568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:46.983310 2568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:46.990967 2568 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:46.991096 2568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:46.991237 2568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.198
I20240502 01:52:47.000165 2568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.31.161.198
--webserver_port=0
--tserver_master_addrs=127.31.161.254:45525
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.198
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:47.001544 2568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:47.003297 2568 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:47.014955 2573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:47.015270 2576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:47.015631 2574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:47.019346 2568 server_base.cc:1034] running on GCE node
I20240502 01:52:47.019896 2568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:47.020855 2568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:47.022065 2568 hybrid_clock.cc:648] HybridClock initialized: now 1714614767022011 us; error 51 us; skew 500 ppm
I20240502 01:52:47.022368 2568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:47.024557 2568 webserver.cc:460] Webserver started at http://127.31.161.198:41715/ using document root <none> and password file <none>
I20240502 01:52:47.025280 2568 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:47.025403 2568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:47.025720 2568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:47.028242 2568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "9a3448a425c041258991dc60dd64ae7d"
format_stamp: "Formatted at 2024-05-02 01:52:47 on dist-test-slave-06l8"
I20240502 01:52:47.028956 2568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "9a3448a425c041258991dc60dd64ae7d"
format_stamp: "Formatted at 2024-05-02 01:52:47 on dist-test-slave-06l8"
I20240502 01:52:47.033883 2568 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20240502 01:52:47.037202 2582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:47.038429 2568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 01:52:47.038578 2568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9a3448a425c041258991dc60dd64ae7d"
format_stamp: "Formatted at 2024-05-02 01:52:47 on dist-test-slave-06l8"
I20240502 01:52:47.038753 2568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:47.071800 2568 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:47.072649 2568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:47.072907 2568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:47.073979 2568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:47.075521 2568 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:47.075619 2568 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:47.075733 2568 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:47.075801 2568 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:47.105595 2568 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.198:37585
I20240502 01:52:47.105660 2695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.198:37585 every 8 connection(s)
I20240502 01:52:47.107007 2568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:52:47.117311 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 2568
I20240502 01:52:47.117501 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 01:52:47.119491 2696 heartbeater.cc:347] Connected to a master server at 127.31.161.254:45525
I20240502 01:52:47.119771 2696 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:47.120671 2696 heartbeater.cc:511] Master 127.31.161.254:45525 requested a full tablet report, sending...
I20240502 01:52:47.121059 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.199:0
--local_ip_for_outbound_sockets=127.31.161.199
--webserver_interface=127.31.161.199
--webserver_port=0
--tserver_master_addrs=127.31.161.254:45525
--builtin_ntp_servers=127.31.161.212:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:52:47.122421 1849 ts_manager.cc:194] Registered new tserver with Master: 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585)
I20240502 01:52:47.123628 1849 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.198:51513
I20240502 01:52:47.154649 2433 heartbeater.cc:503] Master 127.31.161.254:45525 was elected leader, sending a full tablet report...
W20240502 01:52:47.449396 2699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:47.449678 2699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:47.449775 2699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:47.457399 2699 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:47.457528 2699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:47.457659 2699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.199
I20240502 01:52:47.466846 2699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.31.161.199
--webserver_port=0
--tserver_master_addrs=127.31.161.254:45525
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.199
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:47.468122 2699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:47.469846 2699 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:47.483212 2707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:47.483613 2705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:47.484226 2704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:47.484865 2699 server_base.cc:1034] running on GCE node
I20240502 01:52:47.485471 2699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:47.486357 2699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:47.487542 2699 hybrid_clock.cc:648] HybridClock initialized: now 1714614767487498 us; error 37 us; skew 500 ppm
I20240502 01:52:47.487834 2699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:47.491483 2699 webserver.cc:460] Webserver started at http://127.31.161.199:44149/ using document root <none> and password file <none>
I20240502 01:52:47.492180 2699 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:47.492288 2699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:47.492587 2699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:52:47.495086 2699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "163649b5fbb140f18dccdf75f608b0f9"
format_stamp: "Formatted at 2024-05-02 01:52:47 on dist-test-slave-06l8"
I20240502 01:52:47.495790 2699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "163649b5fbb140f18dccdf75f608b0f9"
format_stamp: "Formatted at 2024-05-02 01:52:47 on dist-test-slave-06l8"
I20240502 01:52:47.500661 2699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20240502 01:52:47.503808 2713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:47.505118 2699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:52:47.505261 2699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "163649b5fbb140f18dccdf75f608b0f9"
format_stamp: "Formatted at 2024-05-02 01:52:47 on dist-test-slave-06l8"
I20240502 01:52:47.505448 2699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:47.528541 2699 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:47.529304 2699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:47.529556 2699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:47.530385 2699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:47.531762 2699 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:47.531863 2699 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:47.531965 2699 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:47.532035 2699 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:47.560995 2699 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.199:46381
I20240502 01:52:47.561049 2826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.199:46381 every 8 connection(s)
I20240502 01:52:47.562242 2699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:52:47.568447 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 2699
I20240502 01:52:47.568607 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 01:52:47.574995 2827 heartbeater.cc:347] Connected to a master server at 127.31.161.254:45525
I20240502 01:52:47.575253 2827 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:47.575965 2827 heartbeater.cc:511] Master 127.31.161.254:45525 requested a full tablet report, sending...
I20240502 01:52:47.577276 1849 ts_manager.cc:194] Registered new tserver with Master: 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381)
I20240502 01:52:47.578086 1849 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.199:57503
I20240502 01:52:47.584463 32391 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 01:52:47.601999 1849 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40648:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 01:52:47.645388 2366 tablet_service.cc:1467] Processing CreateTablet for tablet 74753a745d874fcea0fa9dfc4649aa14 (DEFAULT_TABLE table=TestTable [id=21d6c8edffeb445cac75c180ed630ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:47.645388 2498 tablet_service.cc:1467] Processing CreateTablet for tablet 74753a745d874fcea0fa9dfc4649aa14 (DEFAULT_TABLE table=TestTable [id=21d6c8edffeb445cac75c180ed630ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:47.647573 2366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74753a745d874fcea0fa9dfc4649aa14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:47.648383 2498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74753a745d874fcea0fa9dfc4649aa14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:47.648960 2103 tablet_service.cc:1467] Processing CreateTablet for tablet 74753a745d874fcea0fa9dfc4649aa14 (DEFAULT_TABLE table=TestTable [id=21d6c8edffeb445cac75c180ed630ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:47.650460 2103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74753a745d874fcea0fa9dfc4649aa14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:47.655023 1972 tablet_service.cc:1467] Processing CreateTablet for tablet 74753a745d874fcea0fa9dfc4649aa14 (DEFAULT_TABLE table=TestTable [id=21d6c8edffeb445cac75c180ed630ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:47.656627 1972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74753a745d874fcea0fa9dfc4649aa14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:47.669613 2848 tablet_bootstrap.cc:492] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea: Bootstrap starting.
I20240502 01:52:47.673470 2564 heartbeater.cc:503] Master 127.31.161.254:45525 was elected leader, sending a full tablet report...
I20240502 01:52:47.674572 2848 tablet_bootstrap.cc:654] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:47.674479 2630 tablet_service.cc:1467] Processing CreateTablet for tablet 74753a745d874fcea0fa9dfc4649aa14 (DEFAULT_TABLE table=TestTable [id=21d6c8edffeb445cac75c180ed630ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:47.675997 2630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74753a745d874fcea0fa9dfc4649aa14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:47.675937 2848 log.cc:824] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:47.687041 2849 tablet_bootstrap.cc:492] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119: Bootstrap starting.
I20240502 01:52:47.693728 2853 tablet_bootstrap.cc:492] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d: Bootstrap starting.
I20240502 01:52:47.695457 2848 tablet_bootstrap.cc:492] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea: No bootstrap required, opened a new log
I20240502 01:52:47.695427 2852 tablet_bootstrap.cc:492] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879: Bootstrap starting.
I20240502 01:52:47.695750 2848 ts_tablet_manager.cc:1367] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea: Time spent bootstrapping tablet: real 0.026s user 0.016s sys 0.004s
I20240502 01:52:47.697214 2853 tablet_bootstrap.cc:654] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:47.698354 2849 tablet_bootstrap.cc:654] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:47.698477 2853 log.cc:824] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:47.699726 2851 tablet_bootstrap.cc:492] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a: Bootstrap starting.
I20240502 01:52:47.701927 2849 log.cc:824] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:47.702941 2851 tablet_bootstrap.cc:654] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:47.703450 2852 tablet_bootstrap.cc:654] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:47.703958 2853 tablet_bootstrap.cc:492] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d: No bootstrap required, opened a new log
I20240502 01:52:47.704236 2851 log.cc:824] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:47.704282 2853 ts_tablet_manager.cc:1367] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240502 01:52:47.704232 2848 raft_consensus.cc:348] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:47.704651 2852 log.cc:824] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:47.704697 2848 raft_consensus.cc:374] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:47.704803 2848 raft_consensus.cc:729] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269c9061e1a1435e92f79f0b9486dfea, State: Initialized, Role: FOLLOWER
I20240502 01:52:47.705586 2848 consensus_queue.cc:260] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:47.710523 2852 tablet_bootstrap.cc:492] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879: No bootstrap required, opened a new log
I20240502 01:52:47.711337 2852 ts_tablet_manager.cc:1367] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.000s
I20240502 01:52:47.712319 2849 tablet_bootstrap.cc:492] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119: No bootstrap required, opened a new log
I20240502 01:52:47.712615 2849 ts_tablet_manager.cc:1367] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.000s
I20240502 01:52:47.716586 2853 raft_consensus.cc:348] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:47.718621 2853 raft_consensus.cc:374] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:47.718775 2853 raft_consensus.cc:729] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3448a425c041258991dc60dd64ae7d, State: Initialized, Role: FOLLOWER
I20240502 01:52:47.719696 2853 consensus_queue.cc:260] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:47.721442 2848 ts_tablet_manager.cc:1398] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea: Time spent starting tablet: real 0.026s user 0.009s sys 0.016s
I20240502 01:52:47.721868 2851 tablet_bootstrap.cc:492] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a: No bootstrap required, opened a new log
I20240502 01:52:47.722203 2851 ts_tablet_manager.cc:1367] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.007s
I20240502 01:52:47.722098 2849 raft_consensus.cc:348] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:47.722604 2849 raft_consensus.cc:374] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:47.722719 2849 raft_consensus.cc:729] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d99e1fdda704211b3d101f892c57119, State: Initialized, Role: FOLLOWER
I20240502 01:52:47.723413 2849 consensus_queue.cc:260] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:47.725204 2696 heartbeater.cc:503] Master 127.31.161.254:45525 was elected leader, sending a full tablet report...
I20240502 01:52:47.726052 2853 ts_tablet_manager.cc:1398] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d: Time spent starting tablet: real 0.022s user 0.009s sys 0.006s
I20240502 01:52:47.727706 2849 ts_tablet_manager.cc:1398] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20240502 01:52:47.731040 2852 raft_consensus.cc:348] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:47.731357 2852 raft_consensus.cc:374] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:47.731451 2852 raft_consensus.cc:729] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dae5dc94444d5e92e6907c5b637879, State: Initialized, Role: FOLLOWER
I20240502 01:52:47.732039 2852 consensus_queue.cc:260] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:47.733422 2852 ts_tablet_manager.cc:1398] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879: Time spent starting tablet: real 0.022s user 0.010s sys 0.001s
I20240502 01:52:47.733801 2851 raft_consensus.cc:348] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:47.734129 2851 raft_consensus.cc:374] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:47.734243 2851 raft_consensus.cc:729] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43ea75ecb46ee905b781c91aed95a, State: Initialized, Role: FOLLOWER
I20240502 01:52:47.734803 2851 consensus_queue.cc:260] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:47.736359 2851 ts_tablet_manager.cc:1398] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a: Time spent starting tablet: real 0.014s user 0.008s sys 0.001s
W20240502 01:52:47.768399 2170 tablet.cc:2369] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:52:47.782934 2858 raft_consensus.cc:482] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:47.783161 2858 raft_consensus.cc:504] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:47.785106 2858 leader_election.cc:288] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307)
I20240502 01:52:47.801548 2518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74753a745d874fcea0fa9dfc4649aa14" candidate_uuid: "269c9061e1a1435e92f79f0b9486dfea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119" is_pre_election: true
I20240502 01:52:47.802114 2518 raft_consensus.cc:2454] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269c9061e1a1435e92f79f0b9486dfea in term 0.
I20240502 01:52:47.804549 2387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74753a745d874fcea0fa9dfc4649aa14" candidate_uuid: "269c9061e1a1435e92f79f0b9486dfea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a" is_pre_election: true
I20240502 01:52:47.805100 2387 raft_consensus.cc:2454] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269c9061e1a1435e92f79f0b9486dfea in term 0.
I20240502 01:52:47.805676 2057 leader_election.cc:302] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 269c9061e1a1435e92f79f0b9486dfea, 4d99e1fdda704211b3d101f892c57119, 7ef43ea75ecb46ee905b781c91aed95a; no voters:
I20240502 01:52:47.806195 2858 raft_consensus.cc:2781] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:47.806355 2858 raft_consensus.cc:482] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:47.806473 2858 raft_consensus.cc:3037] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:47.806563 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74753a745d874fcea0fa9dfc4649aa14" candidate_uuid: "269c9061e1a1435e92f79f0b9486dfea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879" is_pre_election: true
I20240502 01:52:47.806810 2650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74753a745d874fcea0fa9dfc4649aa14" candidate_uuid: "269c9061e1a1435e92f79f0b9486dfea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3448a425c041258991dc60dd64ae7d" is_pre_election: true
I20240502 01:52:47.807060 1992 raft_consensus.cc:2454] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269c9061e1a1435e92f79f0b9486dfea in term 0.
I20240502 01:52:47.807260 2650 raft_consensus.cc:2454] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269c9061e1a1435e92f79f0b9486dfea in term 0.
W20240502 01:52:47.844552 2038 tablet.cc:2369] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:52:47.859127 2697 tablet.cc:2369] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:52:47.888363 2434 tablet.cc:2369] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:52:47.897773 2858 raft_consensus.cc:504] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:47.898849 2858 leader_election.cc:288] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [CANDIDATE]: Term 1 election: Requested vote from peers 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307)
I20240502 01:52:47.899335 2518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74753a745d874fcea0fa9dfc4649aa14" candidate_uuid: "269c9061e1a1435e92f79f0b9486dfea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119"
I20240502 01:52:47.899369 2387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74753a745d874fcea0fa9dfc4649aa14" candidate_uuid: "269c9061e1a1435e92f79f0b9486dfea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a"
I20240502 01:52:47.899461 2650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74753a745d874fcea0fa9dfc4649aa14" candidate_uuid: "269c9061e1a1435e92f79f0b9486dfea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3448a425c041258991dc60dd64ae7d"
I20240502 01:52:47.899612 2518 raft_consensus.cc:3037] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:47.899677 2650 raft_consensus.cc:3037] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:47.899642 2387 raft_consensus.cc:3037] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:47.899952 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74753a745d874fcea0fa9dfc4649aa14" candidate_uuid: "269c9061e1a1435e92f79f0b9486dfea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879"
I20240502 01:52:47.900206 1992 raft_consensus.cc:3037] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:47.902796 2650 raft_consensus.cc:2454] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269c9061e1a1435e92f79f0b9486dfea in term 1.
I20240502 01:52:47.903450 2518 raft_consensus.cc:2454] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269c9061e1a1435e92f79f0b9486dfea in term 1.
I20240502 01:52:47.903515 2387 raft_consensus.cc:2454] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269c9061e1a1435e92f79f0b9486dfea in term 1.
I20240502 01:52:47.903939 1992 raft_consensus.cc:2454] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 269c9061e1a1435e92f79f0b9486dfea in term 1.
I20240502 01:52:47.903936 2056 leader_election.cc:302] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 269c9061e1a1435e92f79f0b9486dfea, 4d99e1fdda704211b3d101f892c57119, 9a3448a425c041258991dc60dd64ae7d; no voters:
I20240502 01:52:47.904316 2858 raft_consensus.cc:2781] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:47.904735 2858 raft_consensus.cc:686] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [term 1 LEADER]: Becoming Leader. State: Replica: 269c9061e1a1435e92f79f0b9486dfea, State: Running, Role: LEADER
I20240502 01:52:47.905354 2858 consensus_queue.cc:237] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
W20240502 01:52:47.906277 2565 tablet.cc:2369] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:52:47.910539 1849 catalog_manager.cc:5484] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea reported cstate change: term changed from 0 to 1, leader changed from <none> to 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194). New cstate: current_term: 1 leader_uuid: "269c9061e1a1435e92f79f0b9486dfea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:47.937513 32391 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 01:52:47.939639 32391 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 58dae5dc94444d5e92e6907c5b637879 to finish bootstrapping
I20240502 01:52:47.948530 32391 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 269c9061e1a1435e92f79f0b9486dfea to finish bootstrapping
I20240502 01:52:47.955389 32391 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 446c7f9bf31a4acc9dae8daf43330906 to finish bootstrapping
I20240502 01:52:47.964035 32391 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7ef43ea75ecb46ee905b781c91aed95a to finish bootstrapping
I20240502 01:52:47.972414 32391 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4d99e1fdda704211b3d101f892c57119 to finish bootstrapping
I20240502 01:52:47.979239 32391 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9a3448a425c041258991dc60dd64ae7d to finish bootstrapping
I20240502 01:52:47.986063 32391 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 163649b5fbb140f18dccdf75f608b0f9 to finish bootstrapping
I20240502 01:52:47.994364 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 2699
I20240502 01:52:47.995162 2822 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:52:48.074295 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 2699
I20240502 01:52:48.310714 2858 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:48.371001 2858 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:48.378769 2858 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:48.380066 2858 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:51.838990 1848 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40648:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:52:51.850085 2103 tablet_service.cc:1467] Processing CreateTablet for tablet c291b2a5e9d140f881b2bd54f8c9ed19 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9b1fbaeec6af4ae5a31888af58780907]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.850065 1972 tablet_service.cc:1467] Processing CreateTablet for tablet c291b2a5e9d140f881b2bd54f8c9ed19 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9b1fbaeec6af4ae5a31888af58780907]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.850539 2103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c291b2a5e9d140f881b2bd54f8c9ed19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.850539 1972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c291b2a5e9d140f881b2bd54f8c9ed19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.851399 2365 tablet_service.cc:1467] Processing CreateTablet for tablet 4abedcf755774255867d46154f8131c7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9b1fbaeec6af4ae5a31888af58780907]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.851894 2365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4abedcf755774255867d46154f8131c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.852490 1970 tablet_service.cc:1467] Processing CreateTablet for tablet 73f01fd0dc7f488ca215b7f95ea8d988 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9b1fbaeec6af4ae5a31888af58780907]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.852881 1970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73f01fd0dc7f488ca215b7f95ea8d988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.853479 2366 tablet_service.cc:1467] Processing CreateTablet for tablet c291b2a5e9d140f881b2bd54f8c9ed19 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9b1fbaeec6af4ae5a31888af58780907]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.853860 2366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c291b2a5e9d140f881b2bd54f8c9ed19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.857592 1971 tablet_service.cc:1467] Processing CreateTablet for tablet 4abedcf755774255867d46154f8131c7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9b1fbaeec6af4ae5a31888af58780907]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.858070 1971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4abedcf755774255867d46154f8131c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.862253 2102 tablet_service.cc:1467] Processing CreateTablet for tablet 4abedcf755774255867d46154f8131c7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9b1fbaeec6af4ae5a31888af58780907]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.862327 2101 tablet_service.cc:1467] Processing CreateTablet for tablet 73f01fd0dc7f488ca215b7f95ea8d988 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9b1fbaeec6af4ae5a31888af58780907]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.862705 2102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4abedcf755774255867d46154f8131c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.862933 2101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73f01fd0dc7f488ca215b7f95ea8d988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.878048 2896 tablet_bootstrap.cc:492] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a: Bootstrap starting.
I20240502 01:52:51.881214 2896 tablet_bootstrap.cc:654] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.883642 2895 tablet_bootstrap.cc:492] T 73f01fd0dc7f488ca215b7f95ea8d988 P 58dae5dc94444d5e92e6907c5b637879: Bootstrap starting.
I20240502 01:52:51.883889 2894 tablet_bootstrap.cc:492] T c291b2a5e9d140f881b2bd54f8c9ed19 P 269c9061e1a1435e92f79f0b9486dfea: Bootstrap starting.
I20240502 01:52:51.885116 2498 tablet_service.cc:1467] Processing CreateTablet for tablet c291b2a5e9d140f881b2bd54f8c9ed19 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9b1fbaeec6af4ae5a31888af58780907]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.885531 2894 tablet_bootstrap.cc:654] T c291b2a5e9d140f881b2bd54f8c9ed19 P 269c9061e1a1435e92f79f0b9486dfea: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.885564 2498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c291b2a5e9d140f881b2bd54f8c9ed19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.886227 2896 tablet_bootstrap.cc:492] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a: No bootstrap required, opened a new log
I20240502 01:52:51.886372 2896 ts_tablet_manager.cc:1367] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20240502 01:52:51.887310 2896 raft_consensus.cc:348] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:51.887535 2896 raft_consensus.cc:374] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.887651 2896 raft_consensus.cc:729] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43ea75ecb46ee905b781c91aed95a, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.887923 2896 consensus_queue.cc:260] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:51.889703 2629 tablet_service.cc:1467] Processing CreateTablet for tablet 73f01fd0dc7f488ca215b7f95ea8d988 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9b1fbaeec6af4ae5a31888af58780907]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.890174 2629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73f01fd0dc7f488ca215b7f95ea8d988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.890429 2497 tablet_service.cc:1467] Processing CreateTablet for tablet 73f01fd0dc7f488ca215b7f95ea8d988 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9b1fbaeec6af4ae5a31888af58780907]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.890820 2497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73f01fd0dc7f488ca215b7f95ea8d988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.893236 2630 tablet_service.cc:1467] Processing CreateTablet for tablet 4abedcf755774255867d46154f8131c7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9b1fbaeec6af4ae5a31888af58780907]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.893643 2630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4abedcf755774255867d46154f8131c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.903955 2895 tablet_bootstrap.cc:654] T 73f01fd0dc7f488ca215b7f95ea8d988 P 58dae5dc94444d5e92e6907c5b637879: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.904708 2901 tablet_bootstrap.cc:492] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119: Bootstrap starting.
I20240502 01:52:51.904493 2233 tablet_service.cc:1467] Processing CreateTablet for tablet 4abedcf755774255867d46154f8131c7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9b1fbaeec6af4ae5a31888af58780907]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.905500 2232 tablet_service.cc:1467] Processing CreateTablet for tablet 73f01fd0dc7f488ca215b7f95ea8d988 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9b1fbaeec6af4ae5a31888af58780907]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.906539 2901 tablet_bootstrap.cc:654] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.906895 2232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73f01fd0dc7f488ca215b7f95ea8d988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.909494 2900 tablet_bootstrap.cc:492] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9a3448a425c041258991dc60dd64ae7d: Bootstrap starting.
I20240502 01:52:51.910946 2901 tablet_bootstrap.cc:492] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119: No bootstrap required, opened a new log
I20240502 01:52:51.911077 2901 ts_tablet_manager.cc:1367] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 01:52:51.911177 2900 tablet_bootstrap.cc:654] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9a3448a425c041258991dc60dd64ae7d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.905813 2234 tablet_service.cc:1467] Processing CreateTablet for tablet c291b2a5e9d140f881b2bd54f8c9ed19 (DEFAULT_TABLE table=rebalance_test_table_0 [id=9b1fbaeec6af4ae5a31888af58780907]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:51.911944 2234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c291b2a5e9d140f881b2bd54f8c9ed19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.912010 2901 raft_consensus.cc:348] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:51.912226 2901 raft_consensus.cc:374] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.912313 2901 raft_consensus.cc:729] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d99e1fdda704211b3d101f892c57119, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.913501 2894 tablet_bootstrap.cc:492] T c291b2a5e9d140f881b2bd54f8c9ed19 P 269c9061e1a1435e92f79f0b9486dfea: No bootstrap required, opened a new log
I20240502 01:52:51.913662 2894 ts_tablet_manager.cc:1367] T c291b2a5e9d140f881b2bd54f8c9ed19 P 269c9061e1a1435e92f79f0b9486dfea: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.018s
I20240502 01:52:51.914625 2894 raft_consensus.cc:348] T c291b2a5e9d140f881b2bd54f8c9ed19 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:51.914865 2894 raft_consensus.cc:374] T c291b2a5e9d140f881b2bd54f8c9ed19 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.914945 2894 raft_consensus.cc:729] T c291b2a5e9d140f881b2bd54f8c9ed19 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269c9061e1a1435e92f79f0b9486dfea, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.915185 2894 consensus_queue.cc:260] T c291b2a5e9d140f881b2bd54f8c9ed19 P 269c9061e1a1435e92f79f0b9486dfea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:51.916633 2894 ts_tablet_manager.cc:1398] T c291b2a5e9d140f881b2bd54f8c9ed19 P 269c9061e1a1435e92f79f0b9486dfea: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:52:51.916931 2894 tablet_bootstrap.cc:492] T 4abedcf755774255867d46154f8131c7 P 269c9061e1a1435e92f79f0b9486dfea: Bootstrap starting.
I20240502 01:52:51.918537 2894 tablet_bootstrap.cc:654] T 4abedcf755774255867d46154f8131c7 P 269c9061e1a1435e92f79f0b9486dfea: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.919945 2895 tablet_bootstrap.cc:492] T 73f01fd0dc7f488ca215b7f95ea8d988 P 58dae5dc94444d5e92e6907c5b637879: No bootstrap required, opened a new log
I20240502 01:52:51.920315 2233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4abedcf755774255867d46154f8131c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:51.920703 2895 ts_tablet_manager.cc:1367] T 73f01fd0dc7f488ca215b7f95ea8d988 P 58dae5dc94444d5e92e6907c5b637879: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.013s
I20240502 01:52:51.921278 2900 tablet_bootstrap.cc:492] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9a3448a425c041258991dc60dd64ae7d: No bootstrap required, opened a new log
I20240502 01:52:51.921435 2900 ts_tablet_manager.cc:1367] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9a3448a425c041258991dc60dd64ae7d: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20240502 01:52:51.921720 2895 raft_consensus.cc:348] T 73f01fd0dc7f488ca215b7f95ea8d988 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:51.921957 2895 raft_consensus.cc:374] T 73f01fd0dc7f488ca215b7f95ea8d988 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.922063 2895 raft_consensus.cc:729] T 73f01fd0dc7f488ca215b7f95ea8d988 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dae5dc94444d5e92e6907c5b637879, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.922330 2900 raft_consensus.cc:348] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:51.922624 2900 raft_consensus.cc:374] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.922760 2900 raft_consensus.cc:729] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3448a425c041258991dc60dd64ae7d, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.922683 2901 consensus_queue.cc:260] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:51.923139 2900 consensus_queue.cc:260] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9a3448a425c041258991dc60dd64ae7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:51.925218 2905 raft_consensus.cc:482] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:51.922313 2895 consensus_queue.cc:260] T 73f01fd0dc7f488ca215b7f95ea8d988 P 58dae5dc94444d5e92e6907c5b637879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:51.926163 2900 ts_tablet_manager.cc:1398] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9a3448a425c041258991dc60dd64ae7d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:52:51.926509 2900 tablet_bootstrap.cc:492] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d: Bootstrap starting.
I20240502 01:52:51.928198 2900 tablet_bootstrap.cc:654] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.929328 2905 raft_consensus.cc:504] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:51.925376 2901 ts_tablet_manager.cc:1398] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20240502 01:52:51.931428 2901 tablet_bootstrap.cc:492] T c291b2a5e9d140f881b2bd54f8c9ed19 P 4d99e1fdda704211b3d101f892c57119: Bootstrap starting.
I20240502 01:52:51.933153 2901 tablet_bootstrap.cc:654] T c291b2a5e9d140f881b2bd54f8c9ed19 P 4d99e1fdda704211b3d101f892c57119: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.933164 2896 ts_tablet_manager.cc:1398] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a: Time spent starting tablet: real 0.047s user 0.011s sys 0.003s
I20240502 01:52:51.933609 2896 tablet_bootstrap.cc:492] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a: Bootstrap starting.
I20240502 01:52:51.935221 2896 tablet_bootstrap.cc:654] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.947592 2894 tablet_bootstrap.cc:492] T 4abedcf755774255867d46154f8131c7 P 269c9061e1a1435e92f79f0b9486dfea: No bootstrap required, opened a new log
I20240502 01:52:51.947825 2894 ts_tablet_manager.cc:1367] T 4abedcf755774255867d46154f8131c7 P 269c9061e1a1435e92f79f0b9486dfea: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.008s
I20240502 01:52:51.948801 2894 raft_consensus.cc:348] T 4abedcf755774255867d46154f8131c7 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:51.949177 2894 raft_consensus.cc:374] T 4abedcf755774255867d46154f8131c7 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.949271 2894 raft_consensus.cc:729] T 4abedcf755774255867d46154f8131c7 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269c9061e1a1435e92f79f0b9486dfea, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.949555 2894 consensus_queue.cc:260] T 4abedcf755774255867d46154f8131c7 P 269c9061e1a1435e92f79f0b9486dfea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:51.950167 2894 ts_tablet_manager.cc:1398] T 4abedcf755774255867d46154f8131c7 P 269c9061e1a1435e92f79f0b9486dfea: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:52:51.950538 2894 tablet_bootstrap.cc:492] T 73f01fd0dc7f488ca215b7f95ea8d988 P 269c9061e1a1435e92f79f0b9486dfea: Bootstrap starting.
I20240502 01:52:51.952404 2894 tablet_bootstrap.cc:654] T 73f01fd0dc7f488ca215b7f95ea8d988 P 269c9061e1a1435e92f79f0b9486dfea: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.955070 2911 tablet_bootstrap.cc:492] T c291b2a5e9d140f881b2bd54f8c9ed19 P 446c7f9bf31a4acc9dae8daf43330906: Bootstrap starting.
I20240502 01:52:51.959744 2896 tablet_bootstrap.cc:492] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a: No bootstrap required, opened a new log
I20240502 01:52:51.960009 2896 ts_tablet_manager.cc:1367] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.013s
I20240502 01:52:51.960781 2905 leader_election.cc:288] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621), 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307)
I20240502 01:52:51.960908 2896 raft_consensus.cc:348] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:51.961345 2896 raft_consensus.cc:374] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.961467 2896 raft_consensus.cc:729] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43ea75ecb46ee905b781c91aed95a, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.961861 2896 consensus_queue.cc:260] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:51.961880 2901 tablet_bootstrap.cc:492] T c291b2a5e9d140f881b2bd54f8c9ed19 P 4d99e1fdda704211b3d101f892c57119: No bootstrap required, opened a new log
I20240502 01:52:51.962993 2901 ts_tablet_manager.cc:1367] T c291b2a5e9d140f881b2bd54f8c9ed19 P 4d99e1fdda704211b3d101f892c57119: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.004s
I20240502 01:52:51.963977 2896 ts_tablet_manager.cc:1398] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:52:51.963879 2901 raft_consensus.cc:348] T c291b2a5e9d140f881b2bd54f8c9ed19 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:51.964177 2901 raft_consensus.cc:374] T c291b2a5e9d140f881b2bd54f8c9ed19 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.964272 2901 raft_consensus.cc:729] T c291b2a5e9d140f881b2bd54f8c9ed19 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d99e1fdda704211b3d101f892c57119, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.965759 2900 tablet_bootstrap.cc:492] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d: No bootstrap required, opened a new log
I20240502 01:52:51.965951 2900 ts_tablet_manager.cc:1367] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.003s
I20240502 01:52:51.966799 2900 raft_consensus.cc:348] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:51.967279 2900 raft_consensus.cc:374] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.967401 2900 raft_consensus.cc:729] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3448a425c041258991dc60dd64ae7d, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.967653 2911 tablet_bootstrap.cc:654] T c291b2a5e9d140f881b2bd54f8c9ed19 P 446c7f9bf31a4acc9dae8daf43330906: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.967648 2900 consensus_queue.cc:260] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:51.968222 2900 ts_tablet_manager.cc:1398] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:52:51.969102 2911 log.cc:824] T c291b2a5e9d140f881b2bd54f8c9ed19 P 446c7f9bf31a4acc9dae8daf43330906: Log is configured to *not* fsync() on all Append() calls
I20240502 01:52:51.970291 2901 consensus_queue.cc:260] T c291b2a5e9d140f881b2bd54f8c9ed19 P 4d99e1fdda704211b3d101f892c57119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:51.970506 2894 tablet_bootstrap.cc:492] T 73f01fd0dc7f488ca215b7f95ea8d988 P 269c9061e1a1435e92f79f0b9486dfea: No bootstrap required, opened a new log
I20240502 01:52:51.970708 2894 ts_tablet_manager.cc:1367] T 73f01fd0dc7f488ca215b7f95ea8d988 P 269c9061e1a1435e92f79f0b9486dfea: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.004s
I20240502 01:52:51.971629 2894 raft_consensus.cc:348] T 73f01fd0dc7f488ca215b7f95ea8d988 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:51.972013 2894 raft_consensus.cc:374] T 73f01fd0dc7f488ca215b7f95ea8d988 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.972128 2894 raft_consensus.cc:729] T 73f01fd0dc7f488ca215b7f95ea8d988 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269c9061e1a1435e92f79f0b9486dfea, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.972424 2894 consensus_queue.cc:260] T 73f01fd0dc7f488ca215b7f95ea8d988 P 269c9061e1a1435e92f79f0b9486dfea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:51.973146 2894 ts_tablet_manager.cc:1398] T 73f01fd0dc7f488ca215b7f95ea8d988 P 269c9061e1a1435e92f79f0b9486dfea: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:52:51.973853 2901 ts_tablet_manager.cc:1398] T c291b2a5e9d140f881b2bd54f8c9ed19 P 4d99e1fdda704211b3d101f892c57119: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240502 01:52:51.979821 2124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f01fd0dc7f488ca215b7f95ea8d988" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea" is_pre_election: true
I20240502 01:52:51.980259 2124 raft_consensus.cc:2454] T 73f01fd0dc7f488ca215b7f95ea8d988 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 0.
I20240502 01:52:51.981547 2895 ts_tablet_manager.cc:1398] T 73f01fd0dc7f488ca215b7f95ea8d988 P 58dae5dc94444d5e92e6907c5b637879: Time spent starting tablet: real 0.061s user 0.012s sys 0.012s
I20240502 01:52:51.981922 2895 tablet_bootstrap.cc:492] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879: Bootstrap starting.
I20240502 01:52:51.983609 2895 tablet_bootstrap.cc:654] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.984982 2911 tablet_bootstrap.cc:492] T c291b2a5e9d140f881b2bd54f8c9ed19 P 446c7f9bf31a4acc9dae8daf43330906: No bootstrap required, opened a new log
I20240502 01:52:51.987347 2895 tablet_bootstrap.cc:492] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879: No bootstrap required, opened a new log
I20240502 01:52:51.987519 2895 ts_tablet_manager.cc:1367] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240502 01:52:51.989216 2911 ts_tablet_manager.cc:1367] T c291b2a5e9d140f881b2bd54f8c9ed19 P 446c7f9bf31a4acc9dae8daf43330906: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.003s
I20240502 01:52:51.989305 2895 raft_consensus.cc:348] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:51.989570 2895 raft_consensus.cc:374] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:51.989655 2895 raft_consensus.cc:729] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dae5dc94444d5e92e6907c5b637879, State: Initialized, Role: FOLLOWER
I20240502 01:52:51.989920 2895 consensus_queue.cc:260] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:51.990583 2255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f01fd0dc7f488ca215b7f95ea8d988" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906" is_pre_election: true
I20240502 01:52:51.992805 2650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f01fd0dc7f488ca215b7f95ea8d988" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3448a425c041258991dc60dd64ae7d" is_pre_election: true
I20240502 01:52:51.993018 2650 raft_consensus.cc:2454] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 0.
I20240502 01:52:51.992571 2895 ts_tablet_manager.cc:1398] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:52:51.993531 2895 tablet_bootstrap.cc:492] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879: Bootstrap starting.
I20240502 01:52:51.993496 2451 leader_election.cc:302] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 269c9061e1a1435e92f79f0b9486dfea, 4d99e1fdda704211b3d101f892c57119, 9a3448a425c041258991dc60dd64ae7d; no voters:
W20240502 01:52:51.994117 2451 leader_election.cc:341] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:51.994380 2905 raft_consensus.cc:2781] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:51.994493 2905 raft_consensus.cc:482] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:51.994587 2905 raft_consensus.cc:3037] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:51.995482 2895 tablet_bootstrap.cc:654] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:51.996093 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f01fd0dc7f488ca215b7f95ea8d988" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879" is_pre_election: true
I20240502 01:52:51.996328 1992 raft_consensus.cc:2454] T 73f01fd0dc7f488ca215b7f95ea8d988 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 0.
I20240502 01:52:51.997355 2905 raft_consensus.cc:504] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:51.998689 2124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f01fd0dc7f488ca215b7f95ea8d988" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea"
I20240502 01:52:51.998899 2905 leader_election.cc:288] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [CANDIDATE]: Term 1 election: Requested vote from peers 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621), 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307)
I20240502 01:52:51.998945 2124 raft_consensus.cc:3037] T 73f01fd0dc7f488ca215b7f95ea8d988 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:51.999528 2255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f01fd0dc7f488ca215b7f95ea8d988" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906"
I20240502 01:52:51.999637 2650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f01fd0dc7f488ca215b7f95ea8d988" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3448a425c041258991dc60dd64ae7d"
I20240502 01:52:51.999843 2650 raft_consensus.cc:3037] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:52:52.000036 2451 leader_election.cc:341] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:52.000208 2895 tablet_bootstrap.cc:492] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879: No bootstrap required, opened a new log
I20240502 01:52:52.000310 2895 ts_tablet_manager.cc:1367] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240502 01:52:52.000846 2895 raft_consensus.cc:348] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:52.000995 2895 raft_consensus.cc:374] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:52.001048 2895 raft_consensus.cc:729] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dae5dc94444d5e92e6907c5b637879, State: Initialized, Role: FOLLOWER
I20240502 01:52:52.001307 2895 consensus_queue.cc:260] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:52.002022 2895 ts_tablet_manager.cc:1398] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:52.002319 2124 raft_consensus.cc:2454] T 73f01fd0dc7f488ca215b7f95ea8d988 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 1.
I20240502 01:52:52.002466 2650 raft_consensus.cc:2454] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 1.
I20240502 01:52:52.002858 2451 leader_election.cc:302] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 269c9061e1a1435e92f79f0b9486dfea, 4d99e1fdda704211b3d101f892c57119, 9a3448a425c041258991dc60dd64ae7d; no voters: 446c7f9bf31a4acc9dae8daf43330906
I20240502 01:52:52.003201 2905 raft_consensus.cc:2781] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:52.003173 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f01fd0dc7f488ca215b7f95ea8d988" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879"
I20240502 01:52:52.003360 1992 raft_consensus.cc:3037] T 73f01fd0dc7f488ca215b7f95ea8d988 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.003715 2905 raft_consensus.cc:686] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [term 1 LEADER]: Becoming Leader. State: Replica: 4d99e1fdda704211b3d101f892c57119, State: Running, Role: LEADER
I20240502 01:52:52.004232 2905 consensus_queue.cc:237] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:52.005261 2911 raft_consensus.cc:348] T c291b2a5e9d140f881b2bd54f8c9ed19 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:52.005692 2911 raft_consensus.cc:374] T c291b2a5e9d140f881b2bd54f8c9ed19 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:52.005791 2911 raft_consensus.cc:729] T c291b2a5e9d140f881b2bd54f8c9ed19 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446c7f9bf31a4acc9dae8daf43330906, State: Initialized, Role: FOLLOWER
I20240502 01:52:52.005916 1992 raft_consensus.cc:2454] T 73f01fd0dc7f488ca215b7f95ea8d988 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 1.
I20240502 01:52:52.006533 2911 consensus_queue.cc:260] T c291b2a5e9d140f881b2bd54f8c9ed19 P 446c7f9bf31a4acc9dae8daf43330906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:52.008096 2911 ts_tablet_manager.cc:1398] T c291b2a5e9d140f881b2bd54f8c9ed19 P 446c7f9bf31a4acc9dae8daf43330906: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
I20240502 01:52:52.008502 2911 tablet_bootstrap.cc:492] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906: Bootstrap starting.
I20240502 01:52:52.010174 2911 tablet_bootstrap.cc:654] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:52.011857 1850 catalog_manager.cc:5484] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d99e1fdda704211b3d101f892c57119 (127.31.161.197). New cstate: current_term: 1 leader_uuid: "4d99e1fdda704211b3d101f892c57119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:52.012751 2911 tablet_bootstrap.cc:492] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906: No bootstrap required, opened a new log
I20240502 01:52:52.012888 2911 ts_tablet_manager.cc:1367] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:52:52.013726 2911 raft_consensus.cc:348] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:52.013900 2911 raft_consensus.cc:374] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:52.013996 2911 raft_consensus.cc:729] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446c7f9bf31a4acc9dae8daf43330906, State: Initialized, Role: FOLLOWER
I20240502 01:52:52.014238 2911 consensus_queue.cc:260] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:52.014793 2911 ts_tablet_manager.cc:1398] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:52:52.015097 2911 tablet_bootstrap.cc:492] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906: Bootstrap starting.
I20240502 01:52:52.016254 2911 tablet_bootstrap.cc:654] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:52.017935 2911 tablet_bootstrap.cc:492] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906: No bootstrap required, opened a new log
I20240502 01:52:52.018085 2911 ts_tablet_manager.cc:1367] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:52:52.018646 2911 raft_consensus.cc:348] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:52.018877 2911 raft_consensus.cc:374] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:52.018988 2911 raft_consensus.cc:729] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446c7f9bf31a4acc9dae8daf43330906, State: Initialized, Role: FOLLOWER
I20240502 01:52:52.019204 2911 consensus_queue.cc:260] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:52.019624 2911 ts_tablet_manager.cc:1398] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:52:52.064638 2909 raft_consensus.cc:482] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:52.064834 2909 raft_consensus.cc:504] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:52.066922 2909 leader_election.cc:288] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621)
I20240502 01:52:52.077286 2255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abedcf755774255867d46154f8131c7" candidate_uuid: "58dae5dc94444d5e92e6907c5b637879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906" is_pre_election: true
I20240502 01:52:52.077353 2650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abedcf755774255867d46154f8131c7" candidate_uuid: "58dae5dc94444d5e92e6907c5b637879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3448a425c041258991dc60dd64ae7d" is_pre_election: true
I20240502 01:52:52.077584 2650 raft_consensus.cc:2454] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58dae5dc94444d5e92e6907c5b637879 in term 0.
I20240502 01:52:52.077678 2255 raft_consensus.cc:2454] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58dae5dc94444d5e92e6907c5b637879 in term 0.
I20240502 01:52:52.077684 2124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abedcf755774255867d46154f8131c7" candidate_uuid: "58dae5dc94444d5e92e6907c5b637879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea" is_pre_election: true
I20240502 01:52:52.077941 2124 raft_consensus.cc:2454] T 4abedcf755774255867d46154f8131c7 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58dae5dc94444d5e92e6907c5b637879 in term 0.
I20240502 01:52:52.078250 1925 leader_election.cc:302] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 446c7f9bf31a4acc9dae8daf43330906, 58dae5dc94444d5e92e6907c5b637879, 9a3448a425c041258991dc60dd64ae7d; no voters:
I20240502 01:52:52.078548 2387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abedcf755774255867d46154f8131c7" candidate_uuid: "58dae5dc94444d5e92e6907c5b637879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a" is_pre_election: true
I20240502 01:52:52.078698 2909 raft_consensus.cc:2781] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:52.078799 2387 raft_consensus.cc:2454] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58dae5dc94444d5e92e6907c5b637879 in term 0.
I20240502 01:52:52.078842 2909 raft_consensus.cc:482] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:52.078915 2909 raft_consensus.cc:3037] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.080500 2899 raft_consensus.cc:482] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:52.080652 2899 raft_consensus.cc:504] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:52.081054 2909 raft_consensus.cc:504] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:52.081847 2909 leader_election.cc:288] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [CANDIDATE]: Term 1 election: Requested vote from peers 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621)
I20240502 01:52:52.082269 2650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abedcf755774255867d46154f8131c7" candidate_uuid: "58dae5dc94444d5e92e6907c5b637879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3448a425c041258991dc60dd64ae7d"
I20240502 01:52:52.082417 2124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abedcf755774255867d46154f8131c7" candidate_uuid: "58dae5dc94444d5e92e6907c5b637879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea"
I20240502 01:52:52.082487 2650 raft_consensus.cc:3037] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.082616 2124 raft_consensus.cc:3037] T 4abedcf755774255867d46154f8131c7 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.082762 2899 leader_election.cc:288] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987)
I20240502 01:52:52.085126 2650 raft_consensus.cc:2454] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58dae5dc94444d5e92e6907c5b637879 in term 1.
I20240502 01:52:52.085407 2387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abedcf755774255867d46154f8131c7" candidate_uuid: "58dae5dc94444d5e92e6907c5b637879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a"
I20240502 01:52:52.085563 2124 raft_consensus.cc:2454] T 4abedcf755774255867d46154f8131c7 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58dae5dc94444d5e92e6907c5b637879 in term 1.
I20240502 01:52:52.085616 2387 raft_consensus.cc:3037] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.087646 2387 raft_consensus.cc:2454] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58dae5dc94444d5e92e6907c5b637879 in term 1.
I20240502 01:52:52.088414 2255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abedcf755774255867d46154f8131c7" candidate_uuid: "58dae5dc94444d5e92e6907c5b637879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906"
I20240502 01:52:52.088589 2255 raft_consensus.cc:3037] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.089565 1925 leader_election.cc:302] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 269c9061e1a1435e92f79f0b9486dfea, 58dae5dc94444d5e92e6907c5b637879, 9a3448a425c041258991dc60dd64ae7d; no voters:
I20240502 01:52:52.091166 2909 raft_consensus.cc:2781] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:52.091681 2255 raft_consensus.cc:2454] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58dae5dc94444d5e92e6907c5b637879 in term 1.
I20240502 01:52:52.092377 2909 raft_consensus.cc:686] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [term 1 LEADER]: Becoming Leader. State: Replica: 58dae5dc94444d5e92e6907c5b637879, State: Running, Role: LEADER
I20240502 01:52:52.092958 2909 consensus_queue.cc:237] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:52.101807 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c291b2a5e9d140f881b2bd54f8c9ed19" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879" is_pre_election: true
I20240502 01:52:52.102005 1992 raft_consensus.cc:2454] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 0.
I20240502 01:52:52.102699 2124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c291b2a5e9d140f881b2bd54f8c9ed19" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea" is_pre_election: true
I20240502 01:52:52.102931 2124 raft_consensus.cc:2454] T c291b2a5e9d140f881b2bd54f8c9ed19 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 0.
I20240502 01:52:52.103303 2322 leader_election.cc:302] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 269c9061e1a1435e92f79f0b9486dfea, 58dae5dc94444d5e92e6907c5b637879, 7ef43ea75ecb46ee905b781c91aed95a; no voters:
I20240502 01:52:52.103806 2899 raft_consensus.cc:2781] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:52.103905 2899 raft_consensus.cc:482] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:52.104004 2899 raft_consensus.cc:3037] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.106577 2899 raft_consensus.cc:504] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:52.107609 2255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c291b2a5e9d140f881b2bd54f8c9ed19" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906" is_pre_election: true
I20240502 01:52:52.107671 2254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c291b2a5e9d140f881b2bd54f8c9ed19" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906"
I20240502 01:52:52.108259 2899 leader_election.cc:288] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 election: Requested vote from peers 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987)
I20240502 01:52:52.107846 2255 raft_consensus.cc:2454] T c291b2a5e9d140f881b2bd54f8c9ed19 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 0.
I20240502 01:52:52.108577 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c291b2a5e9d140f881b2bd54f8c9ed19" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879"
I20240502 01:52:52.108772 1992 raft_consensus.cc:3037] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.109290 2124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c291b2a5e9d140f881b2bd54f8c9ed19" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea"
I20240502 01:52:52.109481 2124 raft_consensus.cc:3037] T c291b2a5e9d140f881b2bd54f8c9ed19 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.111354 1992 raft_consensus.cc:2454] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 1.
I20240502 01:52:52.112006 2124 raft_consensus.cc:2454] T c291b2a5e9d140f881b2bd54f8c9ed19 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 1.
I20240502 01:52:52.112399 2322 leader_election.cc:302] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 269c9061e1a1435e92f79f0b9486dfea, 58dae5dc94444d5e92e6907c5b637879, 7ef43ea75ecb46ee905b781c91aed95a; no voters: 446c7f9bf31a4acc9dae8daf43330906
I20240502 01:52:52.112744 2899 raft_consensus.cc:2781] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:52.114965 1850 catalog_manager.cc:5484] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193). New cstate: current_term: 1 leader_uuid: "58dae5dc94444d5e92e6907c5b637879" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:52.115361 2518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c291b2a5e9d140f881b2bd54f8c9ed19" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119" is_pre_election: true
I20240502 01:52:52.115612 2518 raft_consensus.cc:2454] T c291b2a5e9d140f881b2bd54f8c9ed19 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 0.
I20240502 01:52:52.115522 2517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c291b2a5e9d140f881b2bd54f8c9ed19" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119"
I20240502 01:52:52.115788 2517 raft_consensus.cc:3037] T c291b2a5e9d140f881b2bd54f8c9ed19 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.118644 2517 raft_consensus.cc:2454] T c291b2a5e9d140f881b2bd54f8c9ed19 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 1.
I20240502 01:52:52.120036 2899 raft_consensus.cc:686] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Becoming Leader. State: Replica: 7ef43ea75ecb46ee905b781c91aed95a, State: Running, Role: LEADER
I20240502 01:52:52.120483 2899 consensus_queue.cc:237] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:52.124042 1850 catalog_manager.cc:5484] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196). New cstate: current_term: 1 leader_uuid: "7ef43ea75ecb46ee905b781c91aed95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: UNKNOWN } } }
W20240502 01:52:52.197588 2302 tablet.cc:2369] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:52:52.246117 2926 raft_consensus.cc:482] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:52.246358 2926 raft_consensus.cc:504] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:52.248360 2926 leader_election.cc:288] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987), 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307)
I20240502 01:52:52.266327 2124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f01fd0dc7f488ca215b7f95ea8d988" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea" is_pre_election: true
I20240502 01:52:52.266731 2124 raft_consensus.cc:2379] T 73f01fd0dc7f488ca215b7f95ea8d988 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 446c7f9bf31a4acc9dae8daf43330906 in current term 1: Already voted for candidate 4d99e1fdda704211b3d101f892c57119 in this term.
I20240502 01:52:52.268577 1992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f01fd0dc7f488ca215b7f95ea8d988" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879" is_pre_election: true
I20240502 01:52:52.268905 1992 raft_consensus.cc:2379] T 73f01fd0dc7f488ca215b7f95ea8d988 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 446c7f9bf31a4acc9dae8daf43330906 in current term 1: Already voted for candidate 4d99e1fdda704211b3d101f892c57119 in this term.
I20240502 01:52:52.272383 2517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f01fd0dc7f488ca215b7f95ea8d988" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119" is_pre_election: true
I20240502 01:52:52.273064 2187 leader_election.cc:302] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 446c7f9bf31a4acc9dae8daf43330906; no voters: 269c9061e1a1435e92f79f0b9486dfea, 4d99e1fdda704211b3d101f892c57119, 58dae5dc94444d5e92e6907c5b637879
I20240502 01:52:52.273586 2926 raft_consensus.cc:3037] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.275527 2650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73f01fd0dc7f488ca215b7f95ea8d988" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3448a425c041258991dc60dd64ae7d" is_pre_election: true
I20240502 01:52:52.275885 2650 raft_consensus.cc:2379] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 446c7f9bf31a4acc9dae8daf43330906 in current term 1: Already voted for candidate 4d99e1fdda704211b3d101f892c57119 in this term.
I20240502 01:52:52.276535 2926 raft_consensus.cc:2726] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:52:52.461192 2905 consensus_queue.cc:1035] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:52.484534 2909 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:52:52.503865 2943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:52.504150 2943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:52.511469 2943 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 01:52:52.551995 2899 consensus_queue.cc:1035] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:52:52.558461 2909 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:52.564356 2909 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:52.565035 2905 consensus_queue.cc:1035] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:52.565632 2899 consensus_queue.cc:1035] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:52.572685 2253 raft_consensus.cc:3037] T c291b2a5e9d140f881b2bd54f8c9ed19 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:52.576032 2941 consensus_queue.cc:1035] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:52.595052 2941 consensus_queue.cc:1035] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:52.610595 2987 consensus_queue.cc:1035] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:52.610883 2905 consensus_queue.cc:1035] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:52.628669 2937 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 288.501 ms
time per row: 0.14425 ms
I20240502 01:52:53.032872 1846 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40648:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:52:53.041771 2497 tablet_service.cc:1467] Processing CreateTablet for tablet 341e0200ea654dcfb91133135b458a16 (DEFAULT_TABLE table=rebalance_test_table_1 [id=db056d9436d84c2e8d32bb9d34b3c3a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:53.042268 2497 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 341e0200ea654dcfb91133135b458a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:53.042277 2366 tablet_service.cc:1467] Processing CreateTablet for tablet 341e0200ea654dcfb91133135b458a16 (DEFAULT_TABLE table=rebalance_test_table_1 [id=db056d9436d84c2e8d32bb9d34b3c3a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:53.042433 2234 tablet_service.cc:1467] Processing CreateTablet for tablet 341e0200ea654dcfb91133135b458a16 (DEFAULT_TABLE table=rebalance_test_table_1 [id=db056d9436d84c2e8d32bb9d34b3c3a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:53.042706 2366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 341e0200ea654dcfb91133135b458a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:53.042695 2629 tablet_service.cc:1467] Processing CreateTablet for tablet 341e0200ea654dcfb91133135b458a16 (DEFAULT_TABLE table=rebalance_test_table_1 [id=db056d9436d84c2e8d32bb9d34b3c3a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:53.042832 2234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 341e0200ea654dcfb91133135b458a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:53.043133 2629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 341e0200ea654dcfb91133135b458a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:53.046490 2365 tablet_service.cc:1467] Processing CreateTablet for tablet 9895c93b27764c22b77087d74e0149c6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=db056d9436d84c2e8d32bb9d34b3c3a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:53.046888 2365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9895c93b27764c22b77087d74e0149c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:53.046958 1970 tablet_service.cc:1467] Processing CreateTablet for tablet 341e0200ea654dcfb91133135b458a16 (DEFAULT_TABLE table=rebalance_test_table_1 [id=db056d9436d84c2e8d32bb9d34b3c3a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:53.047365 1970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 341e0200ea654dcfb91133135b458a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:53.047622 2630 tablet_service.cc:1467] Processing CreateTablet for tablet 9895c93b27764c22b77087d74e0149c6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=db056d9436d84c2e8d32bb9d34b3c3a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:53.047957 2630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9895c93b27764c22b77087d74e0149c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:53.050012 2498 tablet_service.cc:1467] Processing CreateTablet for tablet 9895c93b27764c22b77087d74e0149c6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=db056d9436d84c2e8d32bb9d34b3c3a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:53.050349 2498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9895c93b27764c22b77087d74e0149c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:53.052142 2103 tablet_service.cc:1467] Processing CreateTablet for tablet 9895c93b27764c22b77087d74e0149c6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=db056d9436d84c2e8d32bb9d34b3c3a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:53.052582 2103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9895c93b27764c22b77087d74e0149c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:53.056428 2233 tablet_service.cc:1467] Processing CreateTablet for tablet 9895c93b27764c22b77087d74e0149c6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=db056d9436d84c2e8d32bb9d34b3c3a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:53.056509 2232 tablet_service.cc:1467] Processing CreateTablet for tablet d592edf3298942cd93739aa846f867a4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=db056d9436d84c2e8d32bb9d34b3c3a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:53.056820 2233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9895c93b27764c22b77087d74e0149c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:53.057257 2364 tablet_service.cc:1467] Processing CreateTablet for tablet d592edf3298942cd93739aa846f867a4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=db056d9436d84c2e8d32bb9d34b3c3a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:53.057699 2364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d592edf3298942cd93739aa846f867a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:53.061877 1972 tablet_service.cc:1467] Processing CreateTablet for tablet d592edf3298942cd93739aa846f867a4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=db056d9436d84c2e8d32bb9d34b3c3a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:53.062278 1972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d592edf3298942cd93739aa846f867a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:53.066229 2232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d592edf3298942cd93739aa846f867a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:53.066504 2496 tablet_service.cc:1467] Processing CreateTablet for tablet d592edf3298942cd93739aa846f867a4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=db056d9436d84c2e8d32bb9d34b3c3a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:53.066985 2496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d592edf3298942cd93739aa846f867a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:53.073132 2630 tablet_service.cc:1467] Processing CreateTablet for tablet d592edf3298942cd93739aa846f867a4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=db056d9436d84c2e8d32bb9d34b3c3a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:53.073534 2630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d592edf3298942cd93739aa846f867a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:53.080566 3044 tablet_bootstrap.cc:492] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906: Bootstrap starting.
I20240502 01:52:53.080644 3049 tablet_bootstrap.cc:492] T 9895c93b27764c22b77087d74e0149c6 P 7ef43ea75ecb46ee905b781c91aed95a: Bootstrap starting.
I20240502 01:52:53.080884 3046 tablet_bootstrap.cc:492] T 341e0200ea654dcfb91133135b458a16 P 58dae5dc94444d5e92e6907c5b637879: Bootstrap starting.
I20240502 01:52:53.081537 3045 tablet_bootstrap.cc:492] T 9895c93b27764c22b77087d74e0149c6 P 269c9061e1a1435e92f79f0b9486dfea: Bootstrap starting.
I20240502 01:52:53.081715 3047 tablet_bootstrap.cc:492] T 9895c93b27764c22b77087d74e0149c6 P 4d99e1fdda704211b3d101f892c57119: Bootstrap starting.
I20240502 01:52:53.082657 3046 tablet_bootstrap.cc:654] T 341e0200ea654dcfb91133135b458a16 P 58dae5dc94444d5e92e6907c5b637879: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:53.082796 3044 tablet_bootstrap.cc:654] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:53.083444 3049 tablet_bootstrap.cc:654] T 9895c93b27764c22b77087d74e0149c6 P 7ef43ea75ecb46ee905b781c91aed95a: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:53.083909 3045 tablet_bootstrap.cc:654] T 9895c93b27764c22b77087d74e0149c6 P 269c9061e1a1435e92f79f0b9486dfea: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:53.085290 3047 tablet_bootstrap.cc:654] T 9895c93b27764c22b77087d74e0149c6 P 4d99e1fdda704211b3d101f892c57119: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:53.088914 3049 tablet_bootstrap.cc:492] T 9895c93b27764c22b77087d74e0149c6 P 7ef43ea75ecb46ee905b781c91aed95a: No bootstrap required, opened a new log
I20240502 01:52:53.089143 3049 ts_tablet_manager.cc:1367] T 9895c93b27764c22b77087d74e0149c6 P 7ef43ea75ecb46ee905b781c91aed95a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 01:52:53.090061 3049 raft_consensus.cc:348] T 9895c93b27764c22b77087d74e0149c6 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:53.090446 3049 raft_consensus.cc:374] T 9895c93b27764c22b77087d74e0149c6 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:53.090581 3049 raft_consensus.cc:729] T 9895c93b27764c22b77087d74e0149c6 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43ea75ecb46ee905b781c91aed95a, State: Initialized, Role: FOLLOWER
I20240502 01:52:53.090876 3049 consensus_queue.cc:260] T 9895c93b27764c22b77087d74e0149c6 P 7ef43ea75ecb46ee905b781c91aed95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:53.091665 3049 ts_tablet_manager.cc:1398] T 9895c93b27764c22b77087d74e0149c6 P 7ef43ea75ecb46ee905b781c91aed95a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:53.091758 3046 tablet_bootstrap.cc:492] T 341e0200ea654dcfb91133135b458a16 P 58dae5dc94444d5e92e6907c5b637879: No bootstrap required, opened a new log
I20240502 01:52:53.091935 3046 ts_tablet_manager.cc:1367] T 341e0200ea654dcfb91133135b458a16 P 58dae5dc94444d5e92e6907c5b637879: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240502 01:52:53.092007 3049 tablet_bootstrap.cc:492] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a: Bootstrap starting.
I20240502 01:52:53.092722 3046 raft_consensus.cc:348] T 341e0200ea654dcfb91133135b458a16 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.093014 3046 raft_consensus.cc:374] T 341e0200ea654dcfb91133135b458a16 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:53.093178 3046 raft_consensus.cc:729] T 341e0200ea654dcfb91133135b458a16 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dae5dc94444d5e92e6907c5b637879, State: Initialized, Role: FOLLOWER
I20240502 01:52:53.093465 3049 tablet_bootstrap.cc:654] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:53.093494 3046 consensus_queue.cc:260] T 341e0200ea654dcfb91133135b458a16 P 58dae5dc94444d5e92e6907c5b637879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.093825 3045 tablet_bootstrap.cc:492] T 9895c93b27764c22b77087d74e0149c6 P 269c9061e1a1435e92f79f0b9486dfea: No bootstrap required, opened a new log
I20240502 01:52:53.093991 3045 ts_tablet_manager.cc:1367] T 9895c93b27764c22b77087d74e0149c6 P 269c9061e1a1435e92f79f0b9486dfea: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.005s
I20240502 01:52:53.094054 3046 ts_tablet_manager.cc:1398] T 341e0200ea654dcfb91133135b458a16 P 58dae5dc94444d5e92e6907c5b637879: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:52:53.094367 3046 tablet_bootstrap.cc:492] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879: Bootstrap starting.
I20240502 01:52:53.094588 3043 tablet_bootstrap.cc:492] T 341e0200ea654dcfb91133135b458a16 P 9a3448a425c041258991dc60dd64ae7d: Bootstrap starting.
I20240502 01:52:53.095850 3046 tablet_bootstrap.cc:654] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:53.096702 3044 tablet_bootstrap.cc:492] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906: No bootstrap required, opened a new log
I20240502 01:52:53.096901 3044 ts_tablet_manager.cc:1367] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20240502 01:52:53.097513 3047 tablet_bootstrap.cc:492] T 9895c93b27764c22b77087d74e0149c6 P 4d99e1fdda704211b3d101f892c57119: No bootstrap required, opened a new log
I20240502 01:52:53.097733 3047 ts_tablet_manager.cc:1367] T 9895c93b27764c22b77087d74e0149c6 P 4d99e1fdda704211b3d101f892c57119: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240502 01:52:53.097738 3044 raft_consensus.cc:348] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.098053 3044 raft_consensus.cc:374] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:53.098418 3045 raft_consensus.cc:348] T 9895c93b27764c22b77087d74e0149c6 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:53.098719 3045 raft_consensus.cc:374] T 9895c93b27764c22b77087d74e0149c6 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:53.098858 3045 raft_consensus.cc:729] T 9895c93b27764c22b77087d74e0149c6 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269c9061e1a1435e92f79f0b9486dfea, State: Initialized, Role: FOLLOWER
I20240502 01:52:53.098189 3044 raft_consensus.cc:729] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446c7f9bf31a4acc9dae8daf43330906, State: Initialized, Role: FOLLOWER
I20240502 01:52:53.099048 3047 raft_consensus.cc:348] T 9895c93b27764c22b77087d74e0149c6 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:53.099292 3047 raft_consensus.cc:374] T 9895c93b27764c22b77087d74e0149c6 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:53.099175 3045 consensus_queue.cc:260] T 9895c93b27764c22b77087d74e0149c6 P 269c9061e1a1435e92f79f0b9486dfea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:53.099426 3047 raft_consensus.cc:729] T 9895c93b27764c22b77087d74e0149c6 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d99e1fdda704211b3d101f892c57119, State: Initialized, Role: FOLLOWER
I20240502 01:52:53.099535 3044 consensus_queue.cc:260] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.099736 3047 consensus_queue.cc:260] T 9895c93b27764c22b77087d74e0149c6 P 4d99e1fdda704211b3d101f892c57119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:53.100447 3045 ts_tablet_manager.cc:1398] T 9895c93b27764c22b77087d74e0149c6 P 269c9061e1a1435e92f79f0b9486dfea: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 01:52:53.101212 3044 ts_tablet_manager.cc:1398] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:52:53.101351 3047 ts_tablet_manager.cc:1398] T 9895c93b27764c22b77087d74e0149c6 P 4d99e1fdda704211b3d101f892c57119: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:52:53.101536 3044 tablet_bootstrap.cc:492] T 9895c93b27764c22b77087d74e0149c6 P 446c7f9bf31a4acc9dae8daf43330906: Bootstrap starting.
I20240502 01:52:53.101747 3047 tablet_bootstrap.cc:492] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119: Bootstrap starting.
I20240502 01:52:53.103652 3044 tablet_bootstrap.cc:654] T 9895c93b27764c22b77087d74e0149c6 P 446c7f9bf31a4acc9dae8daf43330906: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:53.111091 3047 tablet_bootstrap.cc:654] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:53.111478 3043 tablet_bootstrap.cc:654] T 341e0200ea654dcfb91133135b458a16 P 9a3448a425c041258991dc60dd64ae7d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:53.112092 3044 tablet_bootstrap.cc:492] T 9895c93b27764c22b77087d74e0149c6 P 446c7f9bf31a4acc9dae8daf43330906: No bootstrap required, opened a new log
I20240502 01:52:53.112298 3044 ts_tablet_manager.cc:1367] T 9895c93b27764c22b77087d74e0149c6 P 446c7f9bf31a4acc9dae8daf43330906: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240502 01:52:53.113034 3044 raft_consensus.cc:348] T 9895c93b27764c22b77087d74e0149c6 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:53.113354 3044 raft_consensus.cc:374] T 9895c93b27764c22b77087d74e0149c6 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:53.113452 3044 raft_consensus.cc:729] T 9895c93b27764c22b77087d74e0149c6 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446c7f9bf31a4acc9dae8daf43330906, State: Initialized, Role: FOLLOWER
I20240502 01:52:53.113694 3044 consensus_queue.cc:260] T 9895c93b27764c22b77087d74e0149c6 P 446c7f9bf31a4acc9dae8daf43330906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:53.114655 3044 ts_tablet_manager.cc:1398] T 9895c93b27764c22b77087d74e0149c6 P 446c7f9bf31a4acc9dae8daf43330906: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:53.114951 3044 tablet_bootstrap.cc:492] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906: Bootstrap starting.
I20240502 01:52:53.116078 2926 raft_consensus.cc:482] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:53.116416 3044 tablet_bootstrap.cc:654] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:53.116258 2926 raft_consensus.cc:504] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.117791 2926 leader_election.cc:288] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307)
I20240502 01:52:53.118510 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341e0200ea654dcfb91133135b458a16" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879" is_pre_election: true
I20240502 01:52:53.118733 1991 raft_consensus.cc:2454] T 341e0200ea654dcfb91133135b458a16 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 446c7f9bf31a4acc9dae8daf43330906 in term 0.
I20240502 01:52:53.121793 2517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341e0200ea654dcfb91133135b458a16" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119" is_pre_election: true
I20240502 01:52:53.122433 2650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341e0200ea654dcfb91133135b458a16" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3448a425c041258991dc60dd64ae7d" is_pre_election: true
W20240502 01:52:53.124293 2187 leader_election.cc:341] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:52:53.124591 2187 leader_election.cc:341] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:53.129673 3049 tablet_bootstrap.cc:492] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a: No bootstrap required, opened a new log
I20240502 01:52:53.129918 3049 ts_tablet_manager.cc:1367] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.001s
I20240502 01:52:53.130957 3043 tablet_bootstrap.cc:492] T 341e0200ea654dcfb91133135b458a16 P 9a3448a425c041258991dc60dd64ae7d: No bootstrap required, opened a new log
I20240502 01:52:53.131855 3049 raft_consensus.cc:348] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.132288 3043 ts_tablet_manager.cc:1367] T 341e0200ea654dcfb91133135b458a16 P 9a3448a425c041258991dc60dd64ae7d: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20240502 01:52:53.133172 3043 raft_consensus.cc:348] T 341e0200ea654dcfb91133135b458a16 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.133528 3049 raft_consensus.cc:374] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:53.133651 3049 raft_consensus.cc:729] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43ea75ecb46ee905b781c91aed95a, State: Initialized, Role: FOLLOWER
I20240502 01:52:53.133697 3044 tablet_bootstrap.cc:492] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906: No bootstrap required, opened a new log
I20240502 01:52:53.134085 3043 raft_consensus.cc:374] T 341e0200ea654dcfb91133135b458a16 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:53.133987 3049 consensus_queue.cc:260] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.134227 3043 raft_consensus.cc:729] T 341e0200ea654dcfb91133135b458a16 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3448a425c041258991dc60dd64ae7d, State: Initialized, Role: FOLLOWER
I20240502 01:52:53.135044 3043 consensus_queue.cc:260] T 341e0200ea654dcfb91133135b458a16 P 9a3448a425c041258991dc60dd64ae7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.135632 3043 ts_tablet_manager.cc:1398] T 341e0200ea654dcfb91133135b458a16 P 9a3448a425c041258991dc60dd64ae7d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:52:53.136634 3044 ts_tablet_manager.cc:1367] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20240502 01:52:53.136852 3049 ts_tablet_manager.cc:1398] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:52:53.137333 3043 tablet_bootstrap.cc:492] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d: Bootstrap starting.
I20240502 01:52:53.138072 3044 raft_consensus.cc:348] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.138837 3043 tablet_bootstrap.cc:654] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:53.138959 3047 tablet_bootstrap.cc:492] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119: No bootstrap required, opened a new log
I20240502 01:52:53.139137 3047 ts_tablet_manager.cc:1367] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.015s
I20240502 01:52:53.139636 3046 tablet_bootstrap.cc:492] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879: No bootstrap required, opened a new log
I20240502 01:52:53.139866 3046 ts_tablet_manager.cc:1367] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879: Time spent bootstrapping tablet: real 0.046s user 0.012s sys 0.017s
I20240502 01:52:53.139819 3047 raft_consensus.cc:348] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.140071 3047 raft_consensus.cc:374] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:53.140139 3047 raft_consensus.cc:729] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d99e1fdda704211b3d101f892c57119, State: Initialized, Role: FOLLOWER
I20240502 01:52:53.140375 3047 consensus_queue.cc:260] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.140797 3049 tablet_bootstrap.cc:492] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a: Bootstrap starting.
I20240502 01:52:53.140892 3047 ts_tablet_manager.cc:1398] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:52:53.141182 3047 tablet_bootstrap.cc:492] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119: Bootstrap starting.
I20240502 01:52:53.141461 3046 raft_consensus.cc:348] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.141842 3046 raft_consensus.cc:374] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:53.141932 3046 raft_consensus.cc:729] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dae5dc94444d5e92e6907c5b637879, State: Initialized, Role: FOLLOWER
I20240502 01:52:53.142169 3046 consensus_queue.cc:260] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.142711 3047 tablet_bootstrap.cc:654] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:53.143514 3044 raft_consensus.cc:374] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:53.143467 2387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341e0200ea654dcfb91133135b458a16" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a" is_pre_election: true
I20240502 01:52:53.143613 3044 raft_consensus.cc:729] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446c7f9bf31a4acc9dae8daf43330906, State: Initialized, Role: FOLLOWER
I20240502 01:52:53.143659 2387 raft_consensus.cc:2454] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 446c7f9bf31a4acc9dae8daf43330906 in term 0.
I20240502 01:52:53.143828 3044 consensus_queue.cc:260] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.144173 3046 ts_tablet_manager.cc:1398] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:52:53.144381 3044 ts_tablet_manager.cc:1398] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 01:52:53.144520 2188 leader_election.cc:302] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 446c7f9bf31a4acc9dae8daf43330906, 58dae5dc94444d5e92e6907c5b637879, 7ef43ea75ecb46ee905b781c91aed95a; no voters: 4d99e1fdda704211b3d101f892c57119, 9a3448a425c041258991dc60dd64ae7d
I20240502 01:52:53.145027 2926 raft_consensus.cc:2781] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:53.145123 2926 raft_consensus.cc:482] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:53.145212 2926 raft_consensus.cc:3037] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:53.147996 2926 raft_consensus.cc:504] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.149482 2387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341e0200ea654dcfb91133135b458a16" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a"
I20240502 01:52:53.149650 2387 raft_consensus.cc:3037] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:53.150192 2517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341e0200ea654dcfb91133135b458a16" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119"
I20240502 01:52:53.150405 2517 raft_consensus.cc:3037] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:53.152377 2387 raft_consensus.cc:2454] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446c7f9bf31a4acc9dae8daf43330906 in term 1.
I20240502 01:52:53.153355 2926 leader_election.cc:288] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [CANDIDATE]: Term 1 election: Requested vote from peers 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307)
I20240502 01:52:53.153502 2650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341e0200ea654dcfb91133135b458a16" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3448a425c041258991dc60dd64ae7d"
I20240502 01:52:53.153739 2650 raft_consensus.cc:3037] T 341e0200ea654dcfb91133135b458a16 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:53.156822 2650 raft_consensus.cc:2454] T 341e0200ea654dcfb91133135b458a16 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446c7f9bf31a4acc9dae8daf43330906 in term 1.
I20240502 01:52:53.157716 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341e0200ea654dcfb91133135b458a16" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879"
I20240502 01:52:53.157910 1991 raft_consensus.cc:3037] T 341e0200ea654dcfb91133135b458a16 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:53.160727 1991 raft_consensus.cc:2454] T 341e0200ea654dcfb91133135b458a16 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446c7f9bf31a4acc9dae8daf43330906 in term 1.
I20240502 01:52:53.161343 2189 leader_election.cc:302] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 446c7f9bf31a4acc9dae8daf43330906, 58dae5dc94444d5e92e6907c5b637879, 7ef43ea75ecb46ee905b781c91aed95a; no voters:
I20240502 01:52:53.161666 2926 raft_consensus.cc:2781] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:53.163158 2517 raft_consensus.cc:2454] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446c7f9bf31a4acc9dae8daf43330906 in term 1.
I20240502 01:52:53.167188 3049 tablet_bootstrap.cc:654] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:53.169135 2926 raft_consensus.cc:686] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEADER]: Becoming Leader. State: Replica: 446c7f9bf31a4acc9dae8daf43330906, State: Running, Role: LEADER
I20240502 01:52:53.169682 2926 consensus_queue.cc:237] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.170096 3047 tablet_bootstrap.cc:492] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119: No bootstrap required, opened a new log
I20240502 01:52:53.170269 3047 ts_tablet_manager.cc:1367] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.014s
I20240502 01:52:53.172477 3064 raft_consensus.cc:482] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:53.171089 3047 raft_consensus.cc:348] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.172917 3047 raft_consensus.cc:374] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:53.172660 3064 raft_consensus.cc:504] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.173034 3047 raft_consensus.cc:729] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d99e1fdda704211b3d101f892c57119, State: Initialized, Role: FOLLOWER
I20240502 01:52:53.173349 3047 consensus_queue.cc:260] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.173957 3047 ts_tablet_manager.cc:1398] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:52:53.174861 2650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d592edf3298942cd93739aa846f867a4" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3448a425c041258991dc60dd64ae7d" is_pre_election: true
W20240502 01:52:53.175436 2187 leader_election.cc:341] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:53.176192 2387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d592edf3298942cd93739aa846f867a4" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a" is_pre_election: true
W20240502 01:52:53.176847 2188 leader_election.cc:341] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:53.177314 3064 leader_election.cc:288] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307)
I20240502 01:52:53.177625 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d592edf3298942cd93739aa846f867a4" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879" is_pre_election: true
I20240502 01:52:53.177776 3043 tablet_bootstrap.cc:492] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d: No bootstrap required, opened a new log
I20240502 01:52:53.177875 1991 raft_consensus.cc:2454] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 446c7f9bf31a4acc9dae8daf43330906 in term 0.
I20240502 01:52:53.177955 3043 ts_tablet_manager.cc:1367] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d: Time spent bootstrapping tablet: real 0.041s user 0.023s sys 0.010s
I20240502 01:52:53.178365 2517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d592edf3298942cd93739aa846f867a4" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119" is_pre_election: true
I20240502 01:52:53.178581 2517 raft_consensus.cc:2454] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 446c7f9bf31a4acc9dae8daf43330906 in term 0.
I20240502 01:52:53.178813 3043 raft_consensus.cc:348] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:53.179041 2187 leader_election.cc:302] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 446c7f9bf31a4acc9dae8daf43330906, 4d99e1fdda704211b3d101f892c57119, 58dae5dc94444d5e92e6907c5b637879; no voters: 7ef43ea75ecb46ee905b781c91aed95a, 9a3448a425c041258991dc60dd64ae7d
I20240502 01:52:53.179101 3043 raft_consensus.cc:374] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:53.179181 3043 raft_consensus.cc:729] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3448a425c041258991dc60dd64ae7d, State: Initialized, Role: FOLLOWER
I20240502 01:52:53.179553 3064 raft_consensus.cc:2781] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:53.179463 3043 consensus_queue.cc:260] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:53.179663 3064 raft_consensus.cc:482] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:53.179752 3064 raft_consensus.cc:3037] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:53.180074 3043 ts_tablet_manager.cc:1398] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 01:52:53.180402 3043 tablet_bootstrap.cc:492] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d: Bootstrap starting.
I20240502 01:52:53.182013 3043 tablet_bootstrap.cc:654] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:53.183817 3064 raft_consensus.cc:504] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.185447 3049 tablet_bootstrap.cc:492] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a: No bootstrap required, opened a new log
I20240502 01:52:53.185678 3049 ts_tablet_manager.cc:1367] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.001s
I20240502 01:52:53.186563 3049 raft_consensus.cc:348] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.186920 3049 raft_consensus.cc:374] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:53.187067 3049 raft_consensus.cc:729] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43ea75ecb46ee905b781c91aed95a, State: Initialized, Role: FOLLOWER
I20240502 01:52:53.187031 2517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d592edf3298942cd93739aa846f867a4" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119"
I20240502 01:52:53.187201 2517 raft_consensus.cc:3037] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:53.187466 3049 consensus_queue.cc:260] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.188264 3049 ts_tablet_manager.cc:1398] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:53.189810 2517 raft_consensus.cc:2454] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446c7f9bf31a4acc9dae8daf43330906 in term 1.
I20240502 01:52:53.190963 3064 leader_election.cc:288] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [CANDIDATE]: Term 1 election: Requested vote from peers 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307)
I20240502 01:52:53.191305 2650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d592edf3298942cd93739aa846f867a4" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3448a425c041258991dc60dd64ae7d"
W20240502 01:52:53.191824 2187 leader_election.cc:341] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:53.191776 2387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d592edf3298942cd93739aa846f867a4" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a"
I20240502 01:52:53.191964 2387 raft_consensus.cc:3037] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:53.192625 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d592edf3298942cd93739aa846f867a4" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879"
I20240502 01:52:53.192845 1991 raft_consensus.cc:3037] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:53.193320 3043 tablet_bootstrap.cc:492] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d: No bootstrap required, opened a new log
I20240502 01:52:53.192759 1842 catalog_manager.cc:5484] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 reported cstate change: term changed from 0 to 1, leader changed from <none> to 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195). New cstate: current_term: 1 leader_uuid: "446c7f9bf31a4acc9dae8daf43330906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:53.193466 3043 ts_tablet_manager.cc:1367] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20240502 01:52:53.194169 3043 raft_consensus.cc:348] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.194381 3043 raft_consensus.cc:374] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:53.194474 3043 raft_consensus.cc:729] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3448a425c041258991dc60dd64ae7d, State: Initialized, Role: FOLLOWER
I20240502 01:52:53.194691 3043 consensus_queue.cc:260] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.194712 2387 raft_consensus.cc:2454] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446c7f9bf31a4acc9dae8daf43330906 in term 1.
I20240502 01:52:53.195453 1991 raft_consensus.cc:2454] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446c7f9bf31a4acc9dae8daf43330906 in term 1.
I20240502 01:52:53.195799 3043 ts_tablet_manager.cc:1398] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:52:53.196055 2188 leader_election.cc:302] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 446c7f9bf31a4acc9dae8daf43330906, 4d99e1fdda704211b3d101f892c57119, 7ef43ea75ecb46ee905b781c91aed95a; no voters: 9a3448a425c041258991dc60dd64ae7d
I20240502 01:52:53.196373 3064 raft_consensus.cc:2781] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:53.196512 3064 raft_consensus.cc:686] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEADER]: Becoming Leader. State: Replica: 446c7f9bf31a4acc9dae8daf43330906, State: Running, Role: LEADER
I20240502 01:52:53.196832 3064 consensus_queue.cc:237] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.200114 1842 catalog_manager.cc:5484] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 reported cstate change: term changed from 0 to 1, leader changed from <none> to 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195). New cstate: current_term: 1 leader_uuid: "446c7f9bf31a4acc9dae8daf43330906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:53.209592 2998 raft_consensus.cc:482] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:53.209759 2998 raft_consensus.cc:504] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:53.211577 2998 leader_election.cc:288] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987), 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621), 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353)
I20240502 01:52:53.222080 2124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9895c93b27764c22b77087d74e0149c6" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea" is_pre_election: true
I20240502 01:52:53.222321 2124 raft_consensus.cc:2454] T 9895c93b27764c22b77087d74e0149c6 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 0.
I20240502 01:52:53.222363 2387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9895c93b27764c22b77087d74e0149c6" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a" is_pre_election: true
I20240502 01:52:53.222640 2387 raft_consensus.cc:2454] T 9895c93b27764c22b77087d74e0149c6 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 0.
I20240502 01:52:53.223053 2584 leader_election.cc:302] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 269c9061e1a1435e92f79f0b9486dfea, 7ef43ea75ecb46ee905b781c91aed95a, 9a3448a425c041258991dc60dd64ae7d; no voters:
I20240502 01:52:53.223428 2998 raft_consensus.cc:2781] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:53.223387 2517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9895c93b27764c22b77087d74e0149c6" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119" is_pre_election: true
I20240502 01:52:53.223551 2998 raft_consensus.cc:482] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:53.223649 2998 raft_consensus.cc:3037] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:53.223618 2517 raft_consensus.cc:2454] T 9895c93b27764c22b77087d74e0149c6 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 0.
I20240502 01:52:53.226195 2998 raft_consensus.cc:504] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:53.227118 2998 leader_election.cc:288] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [CANDIDATE]: Term 1 election: Requested vote from peers 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987), 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621), 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353)
I20240502 01:52:53.227401 2124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9895c93b27764c22b77087d74e0149c6" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea"
I20240502 01:52:53.227602 2124 raft_consensus.cc:3037] T 9895c93b27764c22b77087d74e0149c6 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:53.227674 2387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9895c93b27764c22b77087d74e0149c6" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a"
I20240502 01:52:53.227895 2387 raft_consensus.cc:3037] T 9895c93b27764c22b77087d74e0149c6 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:53.228329 2517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9895c93b27764c22b77087d74e0149c6" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119"
I20240502 01:52:53.228523 2517 raft_consensus.cc:3037] T 9895c93b27764c22b77087d74e0149c6 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:53.228538 2254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9895c93b27764c22b77087d74e0149c6" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906"
I20240502 01:52:53.228755 2254 raft_consensus.cc:3037] T 9895c93b27764c22b77087d74e0149c6 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:53.230288 2124 raft_consensus.cc:2454] T 9895c93b27764c22b77087d74e0149c6 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 1.
I20240502 01:52:53.230612 2387 raft_consensus.cc:2454] T 9895c93b27764c22b77087d74e0149c6 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 1.
I20240502 01:52:53.230962 2255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9895c93b27764c22b77087d74e0149c6" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906" is_pre_election: true
I20240502 01:52:53.231108 2517 raft_consensus.cc:2454] T 9895c93b27764c22b77087d74e0149c6 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 1.
I20240502 01:52:53.231428 2254 raft_consensus.cc:2454] T 9895c93b27764c22b77087d74e0149c6 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 1.
I20240502 01:52:53.231557 2583 leader_election.cc:302] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 269c9061e1a1435e92f79f0b9486dfea, 4d99e1fdda704211b3d101f892c57119, 9a3448a425c041258991dc60dd64ae7d; no voters:
I20240502 01:52:53.231899 2998 raft_consensus.cc:2781] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:53.232317 2998 raft_consensus.cc:686] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [term 1 LEADER]: Becoming Leader. State: Replica: 9a3448a425c041258991dc60dd64ae7d, State: Running, Role: LEADER
I20240502 01:52:53.232792 2998 consensus_queue.cc:237] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:53.236588 1842 catalog_manager.cc:5484] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3448a425c041258991dc60dd64ae7d (127.31.161.198). New cstate: current_term: 1 leader_uuid: "9a3448a425c041258991dc60dd64ae7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:53.392856 2998 raft_consensus.cc:482] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:53.393162 2998 raft_consensus.cc:504] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } }
I20240502 01:52:53.394604 2998 leader_election.cc:288] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987), 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307)
I20240502 01:52:53.395018 2387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d592edf3298942cd93739aa846f867a4" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a" is_pre_election: true
I20240502 01:52:53.395345 2387 raft_consensus.cc:2379] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3448a425c041258991dc60dd64ae7d in current term 1: Already voted for candidate 446c7f9bf31a4acc9dae8daf43330906 in this term.
I20240502 01:52:53.395678 2254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d592edf3298942cd93739aa846f867a4" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906" is_pre_election: true
I20240502 01:52:53.396323 2517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d592edf3298942cd93739aa846f867a4" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119" is_pre_election: true
I20240502 01:52:53.396698 2517 raft_consensus.cc:2379] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3448a425c041258991dc60dd64ae7d in current term 1: Already voted for candidate 446c7f9bf31a4acc9dae8daf43330906 in this term.
I20240502 01:52:53.397217 2583 leader_election.cc:302] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 9a3448a425c041258991dc60dd64ae7d; no voters: 446c7f9bf31a4acc9dae8daf43330906, 4d99e1fdda704211b3d101f892c57119, 7ef43ea75ecb46ee905b781c91aed95a
I20240502 01:52:53.397526 2998 raft_consensus.cc:3037] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:53.399590 2998 raft_consensus.cc:2726] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:52:53.402046 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d592edf3298942cd93739aa846f867a4" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879" is_pre_election: true
I20240502 01:52:53.402268 1991 raft_consensus.cc:2379] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9a3448a425c041258991dc60dd64ae7d in current term 1: Already voted for candidate 446c7f9bf31a4acc9dae8daf43330906 in this term.
I20240502 01:52:53.556782 3064 consensus_queue.cc:1035] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:52:53.611688 3077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:53.612000 3077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:52:53.612226 3064 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:52:53.620230 3077 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:52:53.640724 2998 consensus_queue.cc:1035] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240502 01:52:53.655759 3064 consensus_queue.cc:1035] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:53.659581 3064 consensus_queue.cc:1035] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:53.688009 2926 consensus_queue.cc:1035] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:53.700515 2998 consensus_queue.cc:1035] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:53.717495 2998 consensus_queue.cc:1035] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:53.719532 3075 consensus_queue.cc:1035] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:53.726365 2926 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:53.754683 3064 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:53.760610 3118 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 343.906 ms
time per row: 0.171953 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 01:52:55.017381 1849 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40648:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:52:55.026811 2365 tablet_service.cc:1467] Processing CreateTablet for tablet 0225b684df4f41499f76fe10b72a8861 (DEFAULT_TABLE table=rebalance_test_table_2 [id=832ef7a5c81541c3a3c9b450b6af8fc1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:55.027288 2365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0225b684df4f41499f76fe10b72a8861. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:55.027222 2103 tablet_service.cc:1467] Processing CreateTablet for tablet 0225b684df4f41499f76fe10b72a8861 (DEFAULT_TABLE table=rebalance_test_table_2 [id=832ef7a5c81541c3a3c9b450b6af8fc1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:55.027297 1970 tablet_service.cc:1467] Processing CreateTablet for tablet 0225b684df4f41499f76fe10b72a8861 (DEFAULT_TABLE table=rebalance_test_table_2 [id=832ef7a5c81541c3a3c9b450b6af8fc1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:55.027709 1970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0225b684df4f41499f76fe10b72a8861. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:55.027734 2103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0225b684df4f41499f76fe10b72a8861. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:55.030658 1971 tablet_service.cc:1467] Processing CreateTablet for tablet 4068710224254825bf42671418592ece (DEFAULT_TABLE table=rebalance_test_table_2 [id=832ef7a5c81541c3a3c9b450b6af8fc1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:55.031083 1971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4068710224254825bf42671418592ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:55.034297 2234 tablet_service.cc:1467] Processing CreateTablet for tablet 677599c7914a4bafb83995cf3240679d (DEFAULT_TABLE table=rebalance_test_table_2 [id=832ef7a5c81541c3a3c9b450b6af8fc1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:55.034597 2629 tablet_service.cc:1467] Processing CreateTablet for tablet 677599c7914a4bafb83995cf3240679d (DEFAULT_TABLE table=rebalance_test_table_2 [id=832ef7a5c81541c3a3c9b450b6af8fc1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:55.034772 2234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 677599c7914a4bafb83995cf3240679d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:55.035035 2629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 677599c7914a4bafb83995cf3240679d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:55.035172 2498 tablet_service.cc:1467] Processing CreateTablet for tablet 0225b684df4f41499f76fe10b72a8861 (DEFAULT_TABLE table=rebalance_test_table_2 [id=832ef7a5c81541c3a3c9b450b6af8fc1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:55.035593 2498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0225b684df4f41499f76fe10b72a8861. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:55.037068 2364 tablet_service.cc:1467] Processing CreateTablet for tablet 677599c7914a4bafb83995cf3240679d (DEFAULT_TABLE table=rebalance_test_table_2 [id=832ef7a5c81541c3a3c9b450b6af8fc1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:55.037490 2364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 677599c7914a4bafb83995cf3240679d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:55.034772 2233 tablet_service.cc:1467] Processing CreateTablet for tablet 4068710224254825bf42671418592ece (DEFAULT_TABLE table=rebalance_test_table_2 [id=832ef7a5c81541c3a3c9b450b6af8fc1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:55.037897 2233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4068710224254825bf42671418592ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:55.039883 2496 tablet_service.cc:1467] Processing CreateTablet for tablet 4068710224254825bf42671418592ece (DEFAULT_TABLE table=rebalance_test_table_2 [id=832ef7a5c81541c3a3c9b450b6af8fc1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:55.040243 2496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4068710224254825bf42671418592ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:55.041126 2630 tablet_service.cc:1467] Processing CreateTablet for tablet 4068710224254825bf42671418592ece (DEFAULT_TABLE table=rebalance_test_table_2 [id=832ef7a5c81541c3a3c9b450b6af8fc1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:55.041524 2630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4068710224254825bf42671418592ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:55.041476 1972 tablet_service.cc:1467] Processing CreateTablet for tablet 677599c7914a4bafb83995cf3240679d (DEFAULT_TABLE table=rebalance_test_table_2 [id=832ef7a5c81541c3a3c9b450b6af8fc1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:55.041846 1972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 677599c7914a4bafb83995cf3240679d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:55.045485 2232 tablet_service.cc:1467] Processing CreateTablet for tablet 0225b684df4f41499f76fe10b72a8861 (DEFAULT_TABLE table=rebalance_test_table_2 [id=832ef7a5c81541c3a3c9b450b6af8fc1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:55.045884 2232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0225b684df4f41499f76fe10b72a8861. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:55.047564 2101 tablet_service.cc:1467] Processing CreateTablet for tablet 4068710224254825bf42671418592ece (DEFAULT_TABLE table=rebalance_test_table_2 [id=832ef7a5c81541c3a3c9b450b6af8fc1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:55.048041 2101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4068710224254825bf42671418592ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:55.049715 2102 tablet_service.cc:1467] Processing CreateTablet for tablet 677599c7914a4bafb83995cf3240679d (DEFAULT_TABLE table=rebalance_test_table_2 [id=832ef7a5c81541c3a3c9b450b6af8fc1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:55.050139 2102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 677599c7914a4bafb83995cf3240679d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:55.060304 3167 tablet_bootstrap.cc:492] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879: Bootstrap starting.
I20240502 01:52:55.062459 3168 tablet_bootstrap.cc:492] T 677599c7914a4bafb83995cf3240679d P 9a3448a425c041258991dc60dd64ae7d: Bootstrap starting.
I20240502 01:52:55.062492 3167 tablet_bootstrap.cc:654] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:55.065088 3170 tablet_bootstrap.cc:492] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a: Bootstrap starting.
I20240502 01:52:55.065405 3168 tablet_bootstrap.cc:654] T 677599c7914a4bafb83995cf3240679d P 9a3448a425c041258991dc60dd64ae7d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:55.068367 3170 tablet_bootstrap.cc:654] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:55.072062 3172 tablet_bootstrap.cc:492] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119: Bootstrap starting.
I20240502 01:52:55.074043 3172 tablet_bootstrap.cc:654] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:55.074149 3170 tablet_bootstrap.cc:492] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a: No bootstrap required, opened a new log
I20240502 01:52:55.074402 3170 ts_tablet_manager.cc:1367] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240502 01:52:55.074824 3169 tablet_bootstrap.cc:492] T 677599c7914a4bafb83995cf3240679d P 446c7f9bf31a4acc9dae8daf43330906: Bootstrap starting.
I20240502 01:52:55.075774 3167 tablet_bootstrap.cc:492] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879: No bootstrap required, opened a new log
I20240502 01:52:55.075287 3170 raft_consensus.cc:348] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:55.076037 3167 ts_tablet_manager.cc:1367] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.004s
I20240502 01:52:55.076120 3170 raft_consensus.cc:374] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:55.076398 3170 raft_consensus.cc:729] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43ea75ecb46ee905b781c91aed95a, State: Initialized, Role: FOLLOWER
I20240502 01:52:55.076505 3169 tablet_bootstrap.cc:654] T 677599c7914a4bafb83995cf3240679d P 446c7f9bf31a4acc9dae8daf43330906: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:55.076864 3167 raft_consensus.cc:348] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:55.076932 3170 consensus_queue.cc:260] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:55.077203 3167 raft_consensus.cc:374] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:55.077369 3167 raft_consensus.cc:729] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dae5dc94444d5e92e6907c5b637879, State: Initialized, Role: FOLLOWER
I20240502 01:52:55.077682 3167 consensus_queue.cc:260] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:55.078241 3170 ts_tablet_manager.cc:1398] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:52:55.078400 3167 ts_tablet_manager.cc:1398] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:52:55.079057 3170 tablet_bootstrap.cc:492] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a: Bootstrap starting.
I20240502 01:52:55.080530 3167 tablet_bootstrap.cc:492] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879: Bootstrap starting.
I20240502 01:52:55.082154 3167 tablet_bootstrap.cc:654] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:55.082737 3170 tablet_bootstrap.cc:654] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:55.087635 3170 tablet_bootstrap.cc:492] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a: No bootstrap required, opened a new log
I20240502 01:52:55.087865 3170 ts_tablet_manager.cc:1367] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240502 01:52:55.087901 3171 tablet_bootstrap.cc:492] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea: Bootstrap starting.
I20240502 01:52:55.088728 3170 raft_consensus.cc:348] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:55.089100 3170 raft_consensus.cc:374] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:55.089224 3170 raft_consensus.cc:729] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43ea75ecb46ee905b781c91aed95a, State: Initialized, Role: FOLLOWER
I20240502 01:52:55.089639 3170 consensus_queue.cc:260] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:55.089866 3171 tablet_bootstrap.cc:654] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:55.090385 3170 ts_tablet_manager.cc:1398] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:52:55.091714 3168 tablet_bootstrap.cc:492] T 677599c7914a4bafb83995cf3240679d P 9a3448a425c041258991dc60dd64ae7d: No bootstrap required, opened a new log
I20240502 01:52:55.091970 3168 ts_tablet_manager.cc:1367] T 677599c7914a4bafb83995cf3240679d P 9a3448a425c041258991dc60dd64ae7d: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.019s
I20240502 01:52:55.092940 3168 raft_consensus.cc:348] T 677599c7914a4bafb83995cf3240679d P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:55.093315 3168 raft_consensus.cc:374] T 677599c7914a4bafb83995cf3240679d P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:55.093451 3168 raft_consensus.cc:729] T 677599c7914a4bafb83995cf3240679d P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3448a425c041258991dc60dd64ae7d, State: Initialized, Role: FOLLOWER
I20240502 01:52:55.093791 3168 consensus_queue.cc:260] T 677599c7914a4bafb83995cf3240679d P 9a3448a425c041258991dc60dd64ae7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:55.094578 3168 ts_tablet_manager.cc:1398] T 677599c7914a4bafb83995cf3240679d P 9a3448a425c041258991dc60dd64ae7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:55.095316 3168 tablet_bootstrap.cc:492] T 4068710224254825bf42671418592ece P 9a3448a425c041258991dc60dd64ae7d: Bootstrap starting.
I20240502 01:52:55.096946 3168 tablet_bootstrap.cc:654] T 4068710224254825bf42671418592ece P 9a3448a425c041258991dc60dd64ae7d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:55.097349 3169 tablet_bootstrap.cc:492] T 677599c7914a4bafb83995cf3240679d P 446c7f9bf31a4acc9dae8daf43330906: No bootstrap required, opened a new log
I20240502 01:52:55.097553 3169 ts_tablet_manager.cc:1367] T 677599c7914a4bafb83995cf3240679d P 446c7f9bf31a4acc9dae8daf43330906: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20240502 01:52:55.098336 3169 raft_consensus.cc:348] T 677599c7914a4bafb83995cf3240679d P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:55.098627 3169 raft_consensus.cc:374] T 677599c7914a4bafb83995cf3240679d P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:55.098758 3169 raft_consensus.cc:729] T 677599c7914a4bafb83995cf3240679d P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446c7f9bf31a4acc9dae8daf43330906, State: Initialized, Role: FOLLOWER
I20240502 01:52:55.099074 3169 consensus_queue.cc:260] T 677599c7914a4bafb83995cf3240679d P 446c7f9bf31a4acc9dae8daf43330906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:55.100641 3172 tablet_bootstrap.cc:492] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119: No bootstrap required, opened a new log
I20240502 01:52:55.100639 3171 tablet_bootstrap.cc:492] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea: No bootstrap required, opened a new log
I20240502 01:52:55.100862 3172 ts_tablet_manager.cc:1367] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.004s
I20240502 01:52:55.101135 3169 ts_tablet_manager.cc:1398] T 677599c7914a4bafb83995cf3240679d P 446c7f9bf31a4acc9dae8daf43330906: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:55.100800 3171 ts_tablet_manager.cc:1367] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240502 01:52:55.101392 3167 tablet_bootstrap.cc:492] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879: No bootstrap required, opened a new log
I20240502 01:52:55.101462 3169 tablet_bootstrap.cc:492] T 4068710224254825bf42671418592ece P 446c7f9bf31a4acc9dae8daf43330906: Bootstrap starting.
I20240502 01:52:55.101619 3167 ts_tablet_manager.cc:1367] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.004s
I20240502 01:52:55.101826 3168 tablet_bootstrap.cc:492] T 4068710224254825bf42671418592ece P 9a3448a425c041258991dc60dd64ae7d: No bootstrap required, opened a new log
I20240502 01:52:55.102022 3171 raft_consensus.cc:348] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:55.102370 3167 raft_consensus.cc:348] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:55.102640 3171 raft_consensus.cc:374] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:55.102674 3167 raft_consensus.cc:374] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:55.102798 3167 raft_consensus.cc:729] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dae5dc94444d5e92e6907c5b637879, State: Initialized, Role: FOLLOWER
I20240502 01:52:55.102803 3171 raft_consensus.cc:729] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269c9061e1a1435e92f79f0b9486dfea, State: Initialized, Role: FOLLOWER
I20240502 01:52:55.102777 3172 raft_consensus.cc:348] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:55.103070 3172 raft_consensus.cc:374] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:55.103206 3172 raft_consensus.cc:729] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d99e1fdda704211b3d101f892c57119, State: Initialized, Role: FOLLOWER
I20240502 01:52:55.103075 3167 consensus_queue.cc:260] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:55.103243 3168 ts_tablet_manager.cc:1367] T 4068710224254825bf42671418592ece P 9a3448a425c041258991dc60dd64ae7d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 01:52:55.103618 3167 ts_tablet_manager.cc:1398] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:52:55.103835 3171 consensus_queue.cc:260] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:55.103958 3168 raft_consensus.cc:348] T 4068710224254825bf42671418592ece P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:55.104269 3168 raft_consensus.cc:374] T 4068710224254825bf42671418592ece P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:55.104386 3168 raft_consensus.cc:729] T 4068710224254825bf42671418592ece P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3448a425c041258991dc60dd64ae7d, State: Initialized, Role: FOLLOWER
I20240502 01:52:55.104569 3172 consensus_queue.cc:260] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:55.104843 3168 consensus_queue.cc:260] T 4068710224254825bf42671418592ece P 9a3448a425c041258991dc60dd64ae7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:55.105104 3167 tablet_bootstrap.cc:492] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879: Bootstrap starting.
I20240502 01:52:55.105258 3171 ts_tablet_manager.cc:1398] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:52:55.105629 3168 ts_tablet_manager.cc:1398] T 4068710224254825bf42671418592ece P 9a3448a425c041258991dc60dd64ae7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:55.106330 3169 tablet_bootstrap.cc:654] T 4068710224254825bf42671418592ece P 446c7f9bf31a4acc9dae8daf43330906: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:55.107192 3167 tablet_bootstrap.cc:654] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:55.109247 2992 raft_consensus.cc:482] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:55.109436 2992 raft_consensus.cc:504] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:55.110630 2992 leader_election.cc:288] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987), 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353)
I20240502 01:52:55.111402 2124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4068710224254825bf42671418592ece" candidate_uuid: "58dae5dc94444d5e92e6907c5b637879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea" is_pre_election: true
W20240502 01:52:55.112387 1928 leader_election.cc:341] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:55.113493 2650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4068710224254825bf42671418592ece" candidate_uuid: "58dae5dc94444d5e92e6907c5b637879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3448a425c041258991dc60dd64ae7d" is_pre_election: true
I20240502 01:52:55.113706 2650 raft_consensus.cc:2454] T 4068710224254825bf42671418592ece P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58dae5dc94444d5e92e6907c5b637879 in term 0.
I20240502 01:52:55.115145 2982 raft_consensus.cc:482] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:55.115337 2982 raft_consensus.cc:504] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:55.116160 3172 ts_tablet_manager.cc:1398] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240502 01:52:55.116430 2879 raft_consensus.cc:482] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:55.116417 2982 leader_election.cc:288] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987)
I20240502 01:52:55.116580 2879 raft_consensus.cc:504] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:55.116905 2254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906" is_pre_election: true
I20240502 01:52:55.117655 2879 leader_election.cc:288] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987)
W20240502 01:52:55.117573 2319 leader_election.cc:341] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:55.118072 3172 tablet_bootstrap.cc:492] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119: Bootstrap starting.
I20240502 01:52:55.119333 2387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861" candidate_uuid: "269c9061e1a1435e92f79f0b9486dfea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a" is_pre_election: true
I20240502 01:52:55.119567 2387 raft_consensus.cc:2454] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 269c9061e1a1435e92f79f0b9486dfea in term 0.
I20240502 01:52:55.121416 3172 tablet_bootstrap.cc:654] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:55.121680 2124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea" is_pre_election: true
I20240502 01:52:55.121877 2124 raft_consensus.cc:2454] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 0.
I20240502 01:52:55.123462 2517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119" is_pre_election: true
I20240502 01:52:55.123699 2517 raft_consensus.cc:2454] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 0.
I20240502 01:52:55.124219 2319 leader_election.cc:302] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 269c9061e1a1435e92f79f0b9486dfea, 4d99e1fdda704211b3d101f892c57119, 7ef43ea75ecb46ee905b781c91aed95a; no voters: 446c7f9bf31a4acc9dae8daf43330906
I20240502 01:52:55.124542 2982 raft_consensus.cc:2781] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:55.124644 2982 raft_consensus.cc:482] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:55.124717 2982 raft_consensus.cc:3037] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:55.125046 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861" candidate_uuid: "269c9061e1a1435e92f79f0b9486dfea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879" is_pre_election: true
I20240502 01:52:55.125658 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879" is_pre_election: true
I20240502 01:52:55.127795 2982 raft_consensus.cc:504] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
W20240502 01:52:55.128363 2321 leader_election.cc:341] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:55.129390 2254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906"
W20240502 01:52:55.129753 2058 leader_election.cc:341] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:52:55.129981 2319 leader_election.cc:341] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:55.129992 2982 leader_election.cc:288] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 election: Requested vote from peers 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987)
I20240502 01:52:55.130385 2124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea"
I20240502 01:52:55.130601 2124 raft_consensus.cc:3037] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:55.130561 2517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119"
I20240502 01:52:55.130761 2517 raft_consensus.cc:3037] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:55.133617 2517 raft_consensus.cc:2454] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 1.
I20240502 01:52:55.134512 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879"
W20240502 01:52:55.136168 2321 leader_election.cc:341] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:55.139382 2124 raft_consensus.cc:2454] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 1.
I20240502 01:52:55.139820 2322 leader_election.cc:302] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 269c9061e1a1435e92f79f0b9486dfea, 4d99e1fdda704211b3d101f892c57119, 7ef43ea75ecb46ee905b781c91aed95a; no voters: 446c7f9bf31a4acc9dae8daf43330906, 58dae5dc94444d5e92e6907c5b637879
I20240502 01:52:55.140322 2982 raft_consensus.cc:2781] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:55.153367 2517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4068710224254825bf42671418592ece" candidate_uuid: "58dae5dc94444d5e92e6907c5b637879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119" is_pre_election: true
W20240502 01:52:55.153956 1925 leader_election.cc:341] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:55.154774 3172 tablet_bootstrap.cc:492] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119: No bootstrap required, opened a new log
I20240502 01:52:55.154987 3172 ts_tablet_manager.cc:1367] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.000s
I20240502 01:52:55.155786 3172 raft_consensus.cc:348] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:55.156093 3172 raft_consensus.cc:374] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:55.156172 3172 raft_consensus.cc:729] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d99e1fdda704211b3d101f892c57119, State: Initialized, Role: FOLLOWER
I20240502 01:52:55.156463 3172 consensus_queue.cc:260] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:55.157231 3172 ts_tablet_manager.cc:1398] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 01:52:55.179908 3171 tablet_bootstrap.cc:492] T 677599c7914a4bafb83995cf3240679d P 269c9061e1a1435e92f79f0b9486dfea: Bootstrap starting.
I20240502 01:52:55.181663 3171 tablet_bootstrap.cc:654] T 677599c7914a4bafb83995cf3240679d P 269c9061e1a1435e92f79f0b9486dfea: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:55.188812 3171 tablet_bootstrap.cc:492] T 677599c7914a4bafb83995cf3240679d P 269c9061e1a1435e92f79f0b9486dfea: No bootstrap required, opened a new log
I20240502 01:52:55.188689 2517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861" candidate_uuid: "269c9061e1a1435e92f79f0b9486dfea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119" is_pre_election: true
I20240502 01:52:55.189023 2517 raft_consensus.cc:2379] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 269c9061e1a1435e92f79f0b9486dfea in current term 1: Already voted for candidate 7ef43ea75ecb46ee905b781c91aed95a in this term.
I20240502 01:52:55.189215 3171 ts_tablet_manager.cc:1367] T 677599c7914a4bafb83995cf3240679d P 269c9061e1a1435e92f79f0b9486dfea: Time spent bootstrapping tablet: real 0.084s user 0.003s sys 0.001s
I20240502 01:52:55.190069 2982 raft_consensus.cc:686] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Becoming Leader. State: Replica: 7ef43ea75ecb46ee905b781c91aed95a, State: Running, Role: LEADER
I20240502 01:52:55.190141 3171 raft_consensus.cc:348] T 677599c7914a4bafb83995cf3240679d P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:55.190544 3171 raft_consensus.cc:374] T 677599c7914a4bafb83995cf3240679d P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:55.190663 3171 raft_consensus.cc:729] T 677599c7914a4bafb83995cf3240679d P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269c9061e1a1435e92f79f0b9486dfea, State: Initialized, Role: FOLLOWER
I20240502 01:52:55.190595 2982 consensus_queue.cc:237] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:55.191079 3171 consensus_queue.cc:260] T 677599c7914a4bafb83995cf3240679d P 269c9061e1a1435e92f79f0b9486dfea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:55.191834 3171 ts_tablet_manager.cc:1398] T 677599c7914a4bafb83995cf3240679d P 269c9061e1a1435e92f79f0b9486dfea: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:52:55.193116 3171 tablet_bootstrap.cc:492] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea: Bootstrap starting.
I20240502 01:52:55.194381 2254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4068710224254825bf42671418592ece" candidate_uuid: "58dae5dc94444d5e92e6907c5b637879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906" is_pre_election: true
I20240502 01:52:55.194947 3171 tablet_bootstrap.cc:654] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea: Neither blocks nor log segments found. Creating new log.
W20240502 01:52:55.195019 1925 leader_election.cc:341] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:55.195197 1925 leader_election.cc:302] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 58dae5dc94444d5e92e6907c5b637879, 9a3448a425c041258991dc60dd64ae7d; no voters: 269c9061e1a1435e92f79f0b9486dfea, 446c7f9bf31a4acc9dae8daf43330906, 4d99e1fdda704211b3d101f892c57119
I20240502 01:52:55.195760 2992 raft_consensus.cc:2726] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:52:55.202138 2254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861" candidate_uuid: "269c9061e1a1435e92f79f0b9486dfea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906" is_pre_election: true
W20240502 01:52:55.202770 2056 leader_election.cc:341] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:52:55.203605 2056 leader_election.cc:302] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 269c9061e1a1435e92f79f0b9486dfea, 7ef43ea75ecb46ee905b781c91aed95a; no voters: 446c7f9bf31a4acc9dae8daf43330906, 4d99e1fdda704211b3d101f892c57119, 58dae5dc94444d5e92e6907c5b637879
I20240502 01:52:55.203809 3171 tablet_bootstrap.cc:492] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea: No bootstrap required, opened a new log
I20240502 01:52:55.203804 3167 tablet_bootstrap.cc:492] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879: No bootstrap required, opened a new log
I20240502 01:52:55.203963 3167 ts_tablet_manager.cc:1367] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879: Time spent bootstrapping tablet: real 0.099s user 0.030s sys 0.058s
I20240502 01:52:55.204777 3169 tablet_bootstrap.cc:492] T 4068710224254825bf42671418592ece P 446c7f9bf31a4acc9dae8daf43330906: No bootstrap required, opened a new log
I20240502 01:52:55.204769 3167 raft_consensus.cc:348] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:55.204972 3169 ts_tablet_manager.cc:1367] T 4068710224254825bf42671418592ece P 446c7f9bf31a4acc9dae8daf43330906: Time spent bootstrapping tablet: real 0.104s user 0.036s sys 0.026s
I20240502 01:52:55.205060 3167 raft_consensus.cc:374] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:55.204746 1849 catalog_manager.cc:5484] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196). New cstate: current_term: 1 leader_uuid: "7ef43ea75ecb46ee905b781c91aed95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:55.205201 3167 raft_consensus.cc:729] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dae5dc94444d5e92e6907c5b637879, State: Initialized, Role: FOLLOWER
I20240502 01:52:55.205430 3167 consensus_queue.cc:260] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:55.205991 3171 ts_tablet_manager.cc:1367] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.006s
I20240502 01:52:55.206015 3167 ts_tablet_manager.cc:1398] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:52:55.203961 2879 raft_consensus.cc:2726] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:52:55.206714 3171 raft_consensus.cc:348] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:55.206962 3171 raft_consensus.cc:374] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:55.207057 3171 raft_consensus.cc:729] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269c9061e1a1435e92f79f0b9486dfea, State: Initialized, Role: FOLLOWER
I20240502 01:52:55.207254 3169 raft_consensus.cc:348] T 4068710224254825bf42671418592ece P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:55.207288 3171 consensus_queue.cc:260] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:55.207492 3169 raft_consensus.cc:374] T 4068710224254825bf42671418592ece P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:55.207569 3169 raft_consensus.cc:729] T 4068710224254825bf42671418592ece P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446c7f9bf31a4acc9dae8daf43330906, State: Initialized, Role: FOLLOWER
I20240502 01:52:55.207836 3171 ts_tablet_manager.cc:1398] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:52:55.207902 3169 consensus_queue.cc:260] T 4068710224254825bf42671418592ece P 446c7f9bf31a4acc9dae8daf43330906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:55.209138 3169 ts_tablet_manager.cc:1398] T 4068710224254825bf42671418592ece P 446c7f9bf31a4acc9dae8daf43330906: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:52:55.209481 3169 tablet_bootstrap.cc:492] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906: Bootstrap starting.
I20240502 01:52:55.211681 3169 tablet_bootstrap.cc:654] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:55.213169 3169 tablet_bootstrap.cc:492] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906: No bootstrap required, opened a new log
I20240502 01:52:55.213284 3169 ts_tablet_manager.cc:1367] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:52:55.213753 3169 raft_consensus.cc:348] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:55.213918 3169 raft_consensus.cc:374] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:55.213989 3169 raft_consensus.cc:729] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446c7f9bf31a4acc9dae8daf43330906, State: Initialized, Role: FOLLOWER
I20240502 01:52:55.214166 3169 consensus_queue.cc:260] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:55.214373 3132 raft_consensus.cc:482] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:55.214532 3132 raft_consensus.cc:504] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:55.214867 3169 ts_tablet_manager.cc:1398] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:52:55.215441 3132 leader_election.cc:288] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987)
I20240502 01:52:55.215720 2387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a" is_pre_election: true
I20240502 01:52:55.215910 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879" is_pre_election: true
I20240502 01:52:55.216135 1991 raft_consensus.cc:2454] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 446c7f9bf31a4acc9dae8daf43330906 in term 0.
I20240502 01:52:55.216575 2124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea" is_pre_election: true
I20240502 01:52:55.216782 2124 raft_consensus.cc:2379] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 446c7f9bf31a4acc9dae8daf43330906 in current term 1: Already voted for candidate 7ef43ea75ecb46ee905b781c91aed95a in this term.
I20240502 01:52:55.217589 2517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119" is_pre_election: true
I20240502 01:52:55.217819 2517 raft_consensus.cc:2379] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 446c7f9bf31a4acc9dae8daf43330906 in current term 1: Already voted for candidate 7ef43ea75ecb46ee905b781c91aed95a in this term.
I20240502 01:52:55.218230 2187 leader_election.cc:302] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 446c7f9bf31a4acc9dae8daf43330906, 58dae5dc94444d5e92e6907c5b637879; no voters: 269c9061e1a1435e92f79f0b9486dfea, 4d99e1fdda704211b3d101f892c57119, 7ef43ea75ecb46ee905b781c91aed95a
I20240502 01:52:55.218700 3132 raft_consensus.cc:3037] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:55.220664 3132 raft_consensus.cc:2726] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:52:55.237511 2987 raft_consensus.cc:482] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:55.237725 2987 raft_consensus.cc:504] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:55.238739 2987 leader_election.cc:288] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353)
I20240502 01:52:55.238987 2124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4068710224254825bf42671418592ece" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea" is_pre_election: true
I20240502 01:52:55.239076 2650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4068710224254825bf42671418592ece" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3448a425c041258991dc60dd64ae7d" is_pre_election: true
I20240502 01:52:55.239255 2124 raft_consensus.cc:2454] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 0.
I20240502 01:52:55.239198 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4068710224254825bf42671418592ece" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879" is_pre_election: true
I20240502 01:52:55.239313 2650 raft_consensus.cc:2454] T 4068710224254825bf42671418592ece P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 0.
I20240502 01:52:55.239442 1991 raft_consensus.cc:2454] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 0.
I20240502 01:52:55.239773 2454 leader_election.cc:302] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 269c9061e1a1435e92f79f0b9486dfea, 4d99e1fdda704211b3d101f892c57119, 9a3448a425c041258991dc60dd64ae7d; no voters:
I20240502 01:52:55.239814 2254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4068710224254825bf42671418592ece" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906" is_pre_election: true
I20240502 01:52:55.240078 2987 raft_consensus.cc:2781] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:55.240077 2254 raft_consensus.cc:2454] T 4068710224254825bf42671418592ece P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 0.
I20240502 01:52:55.240180 2987 raft_consensus.cc:482] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:55.240285 2987 raft_consensus.cc:3037] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:55.242210 2987 raft_consensus.cc:504] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:55.243165 2987 leader_election.cc:288] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [CANDIDATE]: Term 1 election: Requested vote from peers 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353)
I20240502 01:52:55.243248 2124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4068710224254825bf42671418592ece" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea"
I20240502 01:52:55.243283 2650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4068710224254825bf42671418592ece" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3448a425c041258991dc60dd64ae7d"
I20240502 01:52:55.243324 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4068710224254825bf42671418592ece" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879"
I20240502 01:52:55.243439 2124 raft_consensus.cc:3037] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:55.243463 2650 raft_consensus.cc:3037] T 4068710224254825bf42671418592ece P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:55.243521 1991 raft_consensus.cc:3037] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:55.243949 2254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4068710224254825bf42671418592ece" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906"
I20240502 01:52:55.244161 2254 raft_consensus.cc:3037] T 4068710224254825bf42671418592ece P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:55.246192 1991 raft_consensus.cc:2454] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 1.
I20240502 01:52:55.246287 2124 raft_consensus.cc:2454] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 1.
I20240502 01:52:55.246354 2650 raft_consensus.cc:2454] T 4068710224254825bf42671418592ece P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 1.
I20240502 01:52:55.246810 2254 raft_consensus.cc:2454] T 4068710224254825bf42671418592ece P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 1.
I20240502 01:52:55.246820 2453 leader_election.cc:302] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 4d99e1fdda704211b3d101f892c57119, 58dae5dc94444d5e92e6907c5b637879, 9a3448a425c041258991dc60dd64ae7d; no voters:
I20240502 01:52:55.247108 2987 raft_consensus.cc:2781] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:55.247517 2987 raft_consensus.cc:686] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [term 1 LEADER]: Becoming Leader. State: Replica: 4d99e1fdda704211b3d101f892c57119, State: Running, Role: LEADER
I20240502 01:52:55.247793 2987 consensus_queue.cc:237] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } }
I20240502 01:52:55.250706 1849 catalog_manager.cc:5484] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d99e1fdda704211b3d101f892c57119 (127.31.161.197). New cstate: current_term: 1 leader_uuid: "4d99e1fdda704211b3d101f892c57119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:55.356619 2982 raft_consensus.cc:482] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:55.356870 2982 raft_consensus.cc:504] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:55.357947 2982 leader_election.cc:288] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353)
I20240502 01:52:55.358490 2124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677599c7914a4bafb83995cf3240679d" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea" is_pre_election: true
I20240502 01:52:55.358816 2124 raft_consensus.cc:2454] T 677599c7914a4bafb83995cf3240679d P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 0.
I20240502 01:52:55.359004 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677599c7914a4bafb83995cf3240679d" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879" is_pre_election: true
I20240502 01:52:55.359316 1991 raft_consensus.cc:2454] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 0.
I20240502 01:52:55.359798 2254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677599c7914a4bafb83995cf3240679d" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906" is_pre_election: true
I20240502 01:52:55.359823 2321 leader_election.cc:302] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 269c9061e1a1435e92f79f0b9486dfea, 58dae5dc94444d5e92e6907c5b637879, 7ef43ea75ecb46ee905b781c91aed95a; no voters:
I20240502 01:52:55.360042 2254 raft_consensus.cc:2454] T 677599c7914a4bafb83995cf3240679d P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 0.
I20240502 01:52:55.360185 2982 raft_consensus.cc:2781] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:55.360293 2982 raft_consensus.cc:482] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:55.360400 2982 raft_consensus.cc:3037] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:55.362469 2982 raft_consensus.cc:504] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:55.363709 2982 leader_election.cc:288] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 election: Requested vote from peers 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353)
I20240502 01:52:55.363691 2124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677599c7914a4bafb83995cf3240679d" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea"
I20240502 01:52:55.363938 2124 raft_consensus.cc:3037] T 677599c7914a4bafb83995cf3240679d P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:55.364112 2254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677599c7914a4bafb83995cf3240679d" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906"
I20240502 01:52:55.364243 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677599c7914a4bafb83995cf3240679d" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879"
I20240502 01:52:55.364318 2254 raft_consensus.cc:3037] T 677599c7914a4bafb83995cf3240679d P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:55.364461 1991 raft_consensus.cc:3037] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:55.365440 2650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677599c7914a4bafb83995cf3240679d" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3448a425c041258991dc60dd64ae7d" is_pre_election: true
I20240502 01:52:55.365481 2649 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "677599c7914a4bafb83995cf3240679d" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3448a425c041258991dc60dd64ae7d"
I20240502 01:52:55.365687 2650 raft_consensus.cc:2454] T 677599c7914a4bafb83995cf3240679d P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 0.
I20240502 01:52:55.366915 2254 raft_consensus.cc:2454] T 677599c7914a4bafb83995cf3240679d P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 1.
I20240502 01:52:55.366921 2124 raft_consensus.cc:2454] T 677599c7914a4bafb83995cf3240679d P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 1.
I20240502 01:52:55.367357 2319 leader_election.cc:302] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 269c9061e1a1435e92f79f0b9486dfea, 446c7f9bf31a4acc9dae8daf43330906, 7ef43ea75ecb46ee905b781c91aed95a; no voters: 9a3448a425c041258991dc60dd64ae7d
I20240502 01:52:55.367628 2982 raft_consensus.cc:2781] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:55.367811 2982 raft_consensus.cc:686] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Becoming Leader. State: Replica: 7ef43ea75ecb46ee905b781c91aed95a, State: Running, Role: LEADER
I20240502 01:52:55.368139 2982 consensus_queue.cc:237] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:55.369315 1991 raft_consensus.cc:2454] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 1.
I20240502 01:52:55.371544 1849 catalog_manager.cc:5484] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196). New cstate: current_term: 1 leader_uuid: "7ef43ea75ecb46ee905b781c91aed95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:55.579043 2982 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:55.625123 2982 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:55.630551 1991 raft_consensus.cc:3037] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:55.635511 3191 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:55.636989 3190 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:55.671723 2987 consensus_queue.cc:1035] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:55.741844 3197 consensus_queue.cc:1035] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:55.742980 3197 consensus_queue.cc:1035] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:55.750298 2987 consensus_queue.cc:1035] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:52:55.752096 3202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:55.752614 3202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:55.764117 3202 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:52:55.785571 2650 raft_consensus.cc:3037] T 677599c7914a4bafb83995cf3240679d P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:55.788884 3190 consensus_queue.cc:1035] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240502 01:52:55.815629 3190 consensus_queue.cc:1035] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:55.824833 2982 consensus_queue.cc:1035] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:55.832410 3189 consensus_queue.cc:1035] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:52:55.990595 3212 log.cc:924] Time spent T 4068710224254825bf42671418592ece P 446c7f9bf31a4acc9dae8daf43330906: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
INSERT report
rows total: 2000
time total: 326.509 ms
time per row: 0.163255 ms
I20240502 01:52:56.247869 1841 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40648:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:52:56.258119 2103 tablet_service.cc:1467] Processing CreateTablet for tablet 4220d1a63bd74ad0a5cac5a2f1bb988c (DEFAULT_TABLE table=rebalance_test_table_3 [id=3402138c5f114264ac2eb0f198dbe6b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:56.258354 2498 tablet_service.cc:1467] Processing CreateTablet for tablet 4220d1a63bd74ad0a5cac5a2f1bb988c (DEFAULT_TABLE table=rebalance_test_table_3 [id=3402138c5f114264ac2eb0f198dbe6b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:56.258581 2103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4220d1a63bd74ad0a5cac5a2f1bb988c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:56.258827 2498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4220d1a63bd74ad0a5cac5a2f1bb988c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:56.263911 3276 tablet_bootstrap.cc:492] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119: Bootstrap starting.
I20240502 01:52:56.264110 1970 tablet_service.cc:1467] Processing CreateTablet for tablet 4220d1a63bd74ad0a5cac5a2f1bb988c (DEFAULT_TABLE table=rebalance_test_table_3 [id=3402138c5f114264ac2eb0f198dbe6b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:56.264652 1970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4220d1a63bd74ad0a5cac5a2f1bb988c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:56.265100 2629 tablet_service.cc:1467] Processing CreateTablet for tablet 4220d1a63bd74ad0a5cac5a2f1bb988c (DEFAULT_TABLE table=rebalance_test_table_3 [id=3402138c5f114264ac2eb0f198dbe6b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:56.265532 2629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4220d1a63bd74ad0a5cac5a2f1bb988c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:56.265658 3277 tablet_bootstrap.cc:492] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea: Bootstrap starting.
I20240502 01:52:56.265756 3276 tablet_bootstrap.cc:654] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:56.268725 3277 tablet_bootstrap.cc:654] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:56.274946 2365 tablet_service.cc:1467] Processing CreateTablet for tablet 4220d1a63bd74ad0a5cac5a2f1bb988c (DEFAULT_TABLE table=rebalance_test_table_3 [id=3402138c5f114264ac2eb0f198dbe6b3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:56.275429 2365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4220d1a63bd74ad0a5cac5a2f1bb988c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:56.277974 1971 tablet_service.cc:1467] Processing CreateTablet for tablet e4672bc55a874f159bf10855f0698ba1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3402138c5f114264ac2eb0f198dbe6b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:56.278390 1971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4672bc55a874f159bf10855f0698ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:56.281982 2234 tablet_service.cc:1467] Processing CreateTablet for tablet 1983e0a53ee54b62afa16676bf986332 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3402138c5f114264ac2eb0f198dbe6b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:56.282260 3279 tablet_bootstrap.cc:492] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d: Bootstrap starting.
I20240502 01:52:56.282446 2234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1983e0a53ee54b62afa16676bf986332. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:56.282928 2498 tablet_service.cc:1467] Processing CreateTablet for tablet 1983e0a53ee54b62afa16676bf986332 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3402138c5f114264ac2eb0f198dbe6b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:56.283318 2629 tablet_service.cc:1467] Processing CreateTablet for tablet 1983e0a53ee54b62afa16676bf986332 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3402138c5f114264ac2eb0f198dbe6b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:56.283423 2498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1983e0a53ee54b62afa16676bf986332. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:56.283490 2496 tablet_service.cc:1467] Processing CreateTablet for tablet e4672bc55a874f159bf10855f0698ba1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3402138c5f114264ac2eb0f198dbe6b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:56.283740 2629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1983e0a53ee54b62afa16676bf986332. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:56.283900 2496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4672bc55a874f159bf10855f0698ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:56.288259 2630 tablet_service.cc:1467] Processing CreateTablet for tablet e4672bc55a874f159bf10855f0698ba1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3402138c5f114264ac2eb0f198dbe6b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:56.288682 2630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4672bc55a874f159bf10855f0698ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:56.289882 3276 tablet_bootstrap.cc:492] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119: No bootstrap required, opened a new log
I20240502 01:52:56.290035 3276 ts_tablet_manager.cc:1367] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.007s
I20240502 01:52:56.290089 3281 tablet_bootstrap.cc:492] T 1983e0a53ee54b62afa16676bf986332 P 446c7f9bf31a4acc9dae8daf43330906: Bootstrap starting.
I20240502 01:52:56.290975 3280 tablet_bootstrap.cc:492] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879: Bootstrap starting.
I20240502 01:52:56.291342 2364 tablet_service.cc:1467] Processing CreateTablet for tablet 1983e0a53ee54b62afa16676bf986332 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3402138c5f114264ac2eb0f198dbe6b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:56.291497 3276 raft_consensus.cc:348] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:56.291695 3281 tablet_bootstrap.cc:654] T 1983e0a53ee54b62afa16676bf986332 P 446c7f9bf31a4acc9dae8daf43330906: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:56.291729 3276 raft_consensus.cc:374] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:56.291781 2364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1983e0a53ee54b62afa16676bf986332. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:56.291808 3276 raft_consensus.cc:729] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d99e1fdda704211b3d101f892c57119, State: Initialized, Role: FOLLOWER
I20240502 01:52:56.292068 3276 consensus_queue.cc:260] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:56.292663 3276 ts_tablet_manager.cc:1398] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:52:56.292934 3276 tablet_bootstrap.cc:492] T 1983e0a53ee54b62afa16676bf986332 P 4d99e1fdda704211b3d101f892c57119: Bootstrap starting.
I20240502 01:52:56.294246 3279 tablet_bootstrap.cc:654] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:56.294517 3276 tablet_bootstrap.cc:654] T 1983e0a53ee54b62afa16676bf986332 P 4d99e1fdda704211b3d101f892c57119: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:56.295706 3280 tablet_bootstrap.cc:654] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:56.297582 2366 tablet_service.cc:1467] Processing CreateTablet for tablet e4672bc55a874f159bf10855f0698ba1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3402138c5f114264ac2eb0f198dbe6b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:56.298027 2366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4672bc55a874f159bf10855f0698ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:56.304313 3276 tablet_bootstrap.cc:492] T 1983e0a53ee54b62afa16676bf986332 P 4d99e1fdda704211b3d101f892c57119: No bootstrap required, opened a new log
I20240502 01:52:56.304529 3276 ts_tablet_manager.cc:1367] T 1983e0a53ee54b62afa16676bf986332 P 4d99e1fdda704211b3d101f892c57119: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240502 01:52:56.305330 3276 raft_consensus.cc:348] T 1983e0a53ee54b62afa16676bf986332 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } }
I20240502 01:52:56.305694 3276 raft_consensus.cc:374] T 1983e0a53ee54b62afa16676bf986332 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:56.305805 3276 raft_consensus.cc:729] T 1983e0a53ee54b62afa16676bf986332 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d99e1fdda704211b3d101f892c57119, State: Initialized, Role: FOLLOWER
I20240502 01:52:56.306139 3276 consensus_queue.cc:260] T 1983e0a53ee54b62afa16676bf986332 P 4d99e1fdda704211b3d101f892c57119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } }
I20240502 01:52:56.306881 3276 ts_tablet_manager.cc:1398] T 1983e0a53ee54b62afa16676bf986332 P 4d99e1fdda704211b3d101f892c57119: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:56.307140 3280 tablet_bootstrap.cc:492] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879: No bootstrap required, opened a new log
I20240502 01:52:56.307334 3280 ts_tablet_manager.cc:1367] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.006s
I20240502 01:52:56.308166 3280 raft_consensus.cc:348] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:56.308439 3280 raft_consensus.cc:374] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:56.308570 3280 raft_consensus.cc:729] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dae5dc94444d5e92e6907c5b637879, State: Initialized, Role: FOLLOWER
I20240502 01:52:56.308899 3280 consensus_queue.cc:260] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:56.309648 3280 ts_tablet_manager.cc:1398] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:52:56.310724 3279 tablet_bootstrap.cc:492] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d: No bootstrap required, opened a new log
I20240502 01:52:56.310962 3279 ts_tablet_manager.cc:1367] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.005s
I20240502 01:52:56.311905 3279 raft_consensus.cc:348] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:56.312167 3279 raft_consensus.cc:374] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:56.312295 3279 raft_consensus.cc:729] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3448a425c041258991dc60dd64ae7d, State: Initialized, Role: FOLLOWER
I20240502 01:52:56.312556 3279 consensus_queue.cc:260] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:56.313352 3279 ts_tablet_manager.cc:1398] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:56.313690 3279 tablet_bootstrap.cc:492] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d: Bootstrap starting.
I20240502 01:52:56.315438 3279 tablet_bootstrap.cc:654] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:56.329108 3280 tablet_bootstrap.cc:492] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879: Bootstrap starting.
I20240502 01:52:56.330600 3280 tablet_bootstrap.cc:654] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:56.337888 3276 tablet_bootstrap.cc:492] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119: Bootstrap starting.
I20240502 01:52:56.339649 3276 tablet_bootstrap.cc:654] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:56.341326 2103 tablet_service.cc:1467] Processing CreateTablet for tablet 1983e0a53ee54b62afa16676bf986332 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3402138c5f114264ac2eb0f198dbe6b3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:56.341763 2103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1983e0a53ee54b62afa16676bf986332. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:56.345937 2101 tablet_service.cc:1467] Processing CreateTablet for tablet e4672bc55a874f159bf10855f0698ba1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=3402138c5f114264ac2eb0f198dbe6b3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:56.346385 2101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4672bc55a874f159bf10855f0698ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:56.345815 3288 tablet_bootstrap.cc:492] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a: Bootstrap starting.
I20240502 01:52:56.348522 3280 tablet_bootstrap.cc:492] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879: No bootstrap required, opened a new log
I20240502 01:52:56.348752 3280 ts_tablet_manager.cc:1367] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.006s
I20240502 01:52:56.349628 3276 tablet_bootstrap.cc:492] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119: No bootstrap required, opened a new log
I20240502 01:52:56.349779 3276 ts_tablet_manager.cc:1367] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.002s
I20240502 01:52:56.349627 3280 raft_consensus.cc:348] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:56.349993 3280 raft_consensus.cc:374] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:56.350075 3280 raft_consensus.cc:729] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dae5dc94444d5e92e6907c5b637879, State: Initialized, Role: FOLLOWER
I20240502 01:52:56.350324 3280 consensus_queue.cc:260] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:56.350494 3276 raft_consensus.cc:348] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:56.350739 3276 raft_consensus.cc:374] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:56.350833 3276 raft_consensus.cc:729] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d99e1fdda704211b3d101f892c57119, State: Initialized, Role: FOLLOWER
I20240502 01:52:56.351064 3280 ts_tablet_manager.cc:1398] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:56.351089 3276 consensus_queue.cc:260] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:56.351635 3276 ts_tablet_manager.cc:1398] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:52:56.352504 3281 tablet_bootstrap.cc:492] T 1983e0a53ee54b62afa16676bf986332 P 446c7f9bf31a4acc9dae8daf43330906: No bootstrap required, opened a new log
I20240502 01:52:56.352658 3281 ts_tablet_manager.cc:1367] T 1983e0a53ee54b62afa16676bf986332 P 446c7f9bf31a4acc9dae8daf43330906: Time spent bootstrapping tablet: real 0.063s user 0.013s sys 0.018s
I20240502 01:52:56.353204 3277 tablet_bootstrap.cc:492] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea: No bootstrap required, opened a new log
I20240502 01:52:56.353359 3277 ts_tablet_manager.cc:1367] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea: Time spent bootstrapping tablet: real 0.088s user 0.001s sys 0.005s
I20240502 01:52:56.353454 3281 raft_consensus.cc:348] T 1983e0a53ee54b62afa16676bf986332 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } }
I20240502 01:52:56.353686 3281 raft_consensus.cc:374] T 1983e0a53ee54b62afa16676bf986332 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:56.353766 3281 raft_consensus.cc:729] T 1983e0a53ee54b62afa16676bf986332 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446c7f9bf31a4acc9dae8daf43330906, State: Initialized, Role: FOLLOWER
I20240502 01:52:56.353987 3281 consensus_queue.cc:260] T 1983e0a53ee54b62afa16676bf986332 P 446c7f9bf31a4acc9dae8daf43330906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } }
I20240502 01:52:56.354585 3281 ts_tablet_manager.cc:1398] T 1983e0a53ee54b62afa16676bf986332 P 446c7f9bf31a4acc9dae8daf43330906: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:52:56.354135 3277 raft_consensus.cc:348] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:56.356134 3277 raft_consensus.cc:374] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:56.356213 3277 raft_consensus.cc:729] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269c9061e1a1435e92f79f0b9486dfea, State: Initialized, Role: FOLLOWER
I20240502 01:52:56.356417 3277 consensus_queue.cc:260] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:56.357009 3277 ts_tablet_manager.cc:1398] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:52:56.357291 3277 tablet_bootstrap.cc:492] T 1983e0a53ee54b62afa16676bf986332 P 269c9061e1a1435e92f79f0b9486dfea: Bootstrap starting.
I20240502 01:52:56.359331 3277 tablet_bootstrap.cc:654] T 1983e0a53ee54b62afa16676bf986332 P 269c9061e1a1435e92f79f0b9486dfea: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:56.362324 3288 tablet_bootstrap.cc:654] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:56.364215 3277 tablet_bootstrap.cc:492] T 1983e0a53ee54b62afa16676bf986332 P 269c9061e1a1435e92f79f0b9486dfea: No bootstrap required, opened a new log
I20240502 01:52:56.364509 3277 ts_tablet_manager.cc:1367] T 1983e0a53ee54b62afa16676bf986332 P 269c9061e1a1435e92f79f0b9486dfea: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20240502 01:52:56.365614 3277 raft_consensus.cc:348] T 1983e0a53ee54b62afa16676bf986332 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } }
I20240502 01:52:56.365984 3277 raft_consensus.cc:374] T 1983e0a53ee54b62afa16676bf986332 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:56.366070 3279 tablet_bootstrap.cc:492] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d: No bootstrap required, opened a new log
I20240502 01:52:56.366151 3277 raft_consensus.cc:729] T 1983e0a53ee54b62afa16676bf986332 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269c9061e1a1435e92f79f0b9486dfea, State: Initialized, Role: FOLLOWER
I20240502 01:52:56.366305 3279 ts_tablet_manager.cc:1367] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d: Time spent bootstrapping tablet: real 0.053s user 0.018s sys 0.010s
I20240502 01:52:56.366492 3277 consensus_queue.cc:260] T 1983e0a53ee54b62afa16676bf986332 P 269c9061e1a1435e92f79f0b9486dfea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } }
I20240502 01:52:56.367055 3279 raft_consensus.cc:348] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } }
I20240502 01:52:56.367327 3277 ts_tablet_manager.cc:1398] T 1983e0a53ee54b62afa16676bf986332 P 269c9061e1a1435e92f79f0b9486dfea: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:52:56.367395 3279 raft_consensus.cc:374] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:56.367532 3279 raft_consensus.cc:729] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3448a425c041258991dc60dd64ae7d, State: Initialized, Role: FOLLOWER
I20240502 01:52:56.367807 3279 consensus_queue.cc:260] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } }
I20240502 01:52:56.368449 3279 ts_tablet_manager.cc:1398] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:52:56.368816 3279 tablet_bootstrap.cc:492] T e4672bc55a874f159bf10855f0698ba1 P 9a3448a425c041258991dc60dd64ae7d: Bootstrap starting.
I20240502 01:52:56.369215 3277 tablet_bootstrap.cc:492] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea: Bootstrap starting.
I20240502 01:52:56.370357 3279 tablet_bootstrap.cc:654] T e4672bc55a874f159bf10855f0698ba1 P 9a3448a425c041258991dc60dd64ae7d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:56.372238 3277 tablet_bootstrap.cc:654] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:56.377434 3288 tablet_bootstrap.cc:492] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a: No bootstrap required, opened a new log
I20240502 01:52:56.377635 3288 ts_tablet_manager.cc:1367] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.000s
I20240502 01:52:56.377882 3277 tablet_bootstrap.cc:492] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea: No bootstrap required, opened a new log
I20240502 01:52:56.378068 3277 ts_tablet_manager.cc:1367] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240502 01:52:56.378450 3288 raft_consensus.cc:348] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:56.378731 3288 raft_consensus.cc:374] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:56.378862 3288 raft_consensus.cc:729] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43ea75ecb46ee905b781c91aed95a, State: Initialized, Role: FOLLOWER
I20240502 01:52:56.378895 3277 raft_consensus.cc:348] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:56.379247 3277 raft_consensus.cc:374] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:56.379125 3288 consensus_queue.cc:260] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:56.379527 3277 raft_consensus.cc:729] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269c9061e1a1435e92f79f0b9486dfea, State: Initialized, Role: FOLLOWER
I20240502 01:52:56.379755 3288 ts_tablet_manager.cc:1398] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:56.379878 3277 consensus_queue.cc:260] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:56.380079 3288 tablet_bootstrap.cc:492] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a: Bootstrap starting.
I20240502 01:52:56.380606 3277 ts_tablet_manager.cc:1398] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:52:56.381665 3288 tablet_bootstrap.cc:654] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:56.383780 3279 tablet_bootstrap.cc:492] T e4672bc55a874f159bf10855f0698ba1 P 9a3448a425c041258991dc60dd64ae7d: No bootstrap required, opened a new log
I20240502 01:52:56.383975 3279 ts_tablet_manager.cc:1367] T e4672bc55a874f159bf10855f0698ba1 P 9a3448a425c041258991dc60dd64ae7d: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20240502 01:52:56.384909 3279 raft_consensus.cc:348] T e4672bc55a874f159bf10855f0698ba1 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:56.385289 3279 raft_consensus.cc:374] T e4672bc55a874f159bf10855f0698ba1 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:56.385424 3279 raft_consensus.cc:729] T e4672bc55a874f159bf10855f0698ba1 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3448a425c041258991dc60dd64ae7d, State: Initialized, Role: FOLLOWER
I20240502 01:52:56.385706 3279 consensus_queue.cc:260] T e4672bc55a874f159bf10855f0698ba1 P 9a3448a425c041258991dc60dd64ae7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:56.386395 3279 ts_tablet_manager.cc:1398] T e4672bc55a874f159bf10855f0698ba1 P 9a3448a425c041258991dc60dd64ae7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:52:56.390161 3288 tablet_bootstrap.cc:492] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a: No bootstrap required, opened a new log
I20240502 01:52:56.390313 3288 ts_tablet_manager.cc:1367] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240502 01:52:56.390890 3288 raft_consensus.cc:348] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:56.391090 3288 raft_consensus.cc:374] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:56.391160 3288 raft_consensus.cc:729] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43ea75ecb46ee905b781c91aed95a, State: Initialized, Role: FOLLOWER
I20240502 01:52:56.391386 3288 consensus_queue.cc:260] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:56.392016 3288 ts_tablet_manager.cc:1398] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:52:56.392307 3288 tablet_bootstrap.cc:492] T 1983e0a53ee54b62afa16676bf986332 P 7ef43ea75ecb46ee905b781c91aed95a: Bootstrap starting.
I20240502 01:52:56.393622 3191 raft_consensus.cc:482] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:56.393810 3288 tablet_bootstrap.cc:654] T 1983e0a53ee54b62afa16676bf986332 P 7ef43ea75ecb46ee905b781c91aed95a: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:56.393815 3191 raft_consensus.cc:504] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:56.394867 3191 leader_election.cc:288] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621), 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585)
I20240502 01:52:56.395479 2123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4220d1a63bd74ad0a5cac5a2f1bb988c" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea" is_pre_election: true
I20240502 01:52:56.395699 2123 raft_consensus.cc:2454] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 0.
I20240502 01:52:56.395732 2517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4220d1a63bd74ad0a5cac5a2f1bb988c" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119" is_pre_election: true
I20240502 01:52:56.395936 2517 raft_consensus.cc:2454] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 0.
I20240502 01:52:56.396080 2650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4220d1a63bd74ad0a5cac5a2f1bb988c" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3448a425c041258991dc60dd64ae7d" is_pre_election: true
I20240502 01:52:56.396301 2650 raft_consensus.cc:2454] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 0.
I20240502 01:52:56.396443 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4220d1a63bd74ad0a5cac5a2f1bb988c" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879" is_pre_election: true
I20240502 01:52:56.396682 1990 raft_consensus.cc:2454] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 0.
I20240502 01:52:56.397115 2321 leader_election.cc:302] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 269c9061e1a1435e92f79f0b9486dfea, 58dae5dc94444d5e92e6907c5b637879, 7ef43ea75ecb46ee905b781c91aed95a; no voters:
I20240502 01:52:56.397387 3191 raft_consensus.cc:2781] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:56.397492 3191 raft_consensus.cc:482] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:56.397606 3191 raft_consensus.cc:3037] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:56.399091 3288 tablet_bootstrap.cc:492] T 1983e0a53ee54b62afa16676bf986332 P 7ef43ea75ecb46ee905b781c91aed95a: No bootstrap required, opened a new log
I20240502 01:52:56.399250 3288 ts_tablet_manager.cc:1367] T 1983e0a53ee54b62afa16676bf986332 P 7ef43ea75ecb46ee905b781c91aed95a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240502 01:52:56.399979 3288 raft_consensus.cc:348] T 1983e0a53ee54b62afa16676bf986332 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } }
I20240502 01:52:56.400195 3288 raft_consensus.cc:374] T 1983e0a53ee54b62afa16676bf986332 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:56.400194 3191 raft_consensus.cc:504] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:56.400369 3288 raft_consensus.cc:729] T 1983e0a53ee54b62afa16676bf986332 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43ea75ecb46ee905b781c91aed95a, State: Initialized, Role: FOLLOWER
I20240502 01:52:56.400607 3288 consensus_queue.cc:260] T 1983e0a53ee54b62afa16676bf986332 P 7ef43ea75ecb46ee905b781c91aed95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } }
I20240502 01:52:56.401175 3288 ts_tablet_manager.cc:1398] T 1983e0a53ee54b62afa16676bf986332 P 7ef43ea75ecb46ee905b781c91aed95a: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 01:52:56.401225 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4220d1a63bd74ad0a5cac5a2f1bb988c" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879"
I20240502 01:52:56.401446 1990 raft_consensus.cc:3037] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:56.401434 2123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4220d1a63bd74ad0a5cac5a2f1bb988c" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea"
I20240502 01:52:56.401616 2123 raft_consensus.cc:3037] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:56.404140 1990 raft_consensus.cc:2454] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 1.
I20240502 01:52:56.404397 2123 raft_consensus.cc:2454] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 1.
I20240502 01:52:56.404827 2322 leader_election.cc:302] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 269c9061e1a1435e92f79f0b9486dfea, 58dae5dc94444d5e92e6907c5b637879, 7ef43ea75ecb46ee905b781c91aed95a; no voters:
I20240502 01:52:56.405166 3269 raft_consensus.cc:2781] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:56.405305 3269 raft_consensus.cc:686] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Becoming Leader. State: Replica: 7ef43ea75ecb46ee905b781c91aed95a, State: Running, Role: LEADER
I20240502 01:52:56.405293 2650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4220d1a63bd74ad0a5cac5a2f1bb988c" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3448a425c041258991dc60dd64ae7d"
I20240502 01:52:56.405318 2517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4220d1a63bd74ad0a5cac5a2f1bb988c" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119"
I20240502 01:52:56.405500 2650 raft_consensus.cc:3037] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:56.405514 2517 raft_consensus.cc:3037] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:56.405616 3269 consensus_queue.cc:237] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:56.404810 3191 leader_election.cc:288] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 election: Requested vote from peers 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621), 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585)
I20240502 01:52:56.408187 2650 raft_consensus.cc:2454] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 1.
I20240502 01:52:56.408257 2517 raft_consensus.cc:2454] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 1.
I20240502 01:52:56.408607 3112 raft_consensus.cc:482] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:56.408754 3112 raft_consensus.cc:504] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } }
I20240502 01:52:56.409823 3112 leader_election.cc:288] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987), 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621)
I20240502 01:52:56.409555 1841 catalog_manager.cc:5484] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196). New cstate: current_term: 1 leader_uuid: "7ef43ea75ecb46ee905b781c91aed95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:56.410063 2254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1983e0a53ee54b62afa16676bf986332" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906" is_pre_election: true
I20240502 01:52:56.410156 2517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1983e0a53ee54b62afa16676bf986332" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119" is_pre_election: true
I20240502 01:52:56.410333 2254 raft_consensus.cc:2454] T 1983e0a53ee54b62afa16676bf986332 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 0.
I20240502 01:52:56.410362 2517 raft_consensus.cc:2454] T 1983e0a53ee54b62afa16676bf986332 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 0.
I20240502 01:52:56.410740 2387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1983e0a53ee54b62afa16676bf986332" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a" is_pre_election: true
I20240502 01:52:56.410820 2583 leader_election.cc:302] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 446c7f9bf31a4acc9dae8daf43330906, 4d99e1fdda704211b3d101f892c57119, 9a3448a425c041258991dc60dd64ae7d; no voters:
I20240502 01:52:56.410899 2387 raft_consensus.cc:2454] T 1983e0a53ee54b62afa16676bf986332 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 0.
I20240502 01:52:56.411093 3112 raft_consensus.cc:2781] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:56.411204 3112 raft_consensus.cc:482] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:56.411288 3112 raft_consensus.cc:3037] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:56.411401 2123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1983e0a53ee54b62afa16676bf986332" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea" is_pre_election: true
I20240502 01:52:56.411607 2123 raft_consensus.cc:2454] T 1983e0a53ee54b62afa16676bf986332 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 0.
I20240502 01:52:56.414139 3112 raft_consensus.cc:504] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } }
I20240502 01:52:56.415138 2254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1983e0a53ee54b62afa16676bf986332" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906"
I20240502 01:52:56.415160 2387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1983e0a53ee54b62afa16676bf986332" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a"
I20240502 01:52:56.415184 2517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1983e0a53ee54b62afa16676bf986332" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119"
I20240502 01:52:56.415268 2123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1983e0a53ee54b62afa16676bf986332" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea"
I20240502 01:52:56.415350 2387 raft_consensus.cc:3037] T 1983e0a53ee54b62afa16676bf986332 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:56.415365 2254 raft_consensus.cc:3037] T 1983e0a53ee54b62afa16676bf986332 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:56.415400 2517 raft_consensus.cc:3037] T 1983e0a53ee54b62afa16676bf986332 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:56.415446 2123 raft_consensus.cc:3037] T 1983e0a53ee54b62afa16676bf986332 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:56.416766 3112 leader_election.cc:288] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [CANDIDATE]: Term 1 election: Requested vote from peers 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987), 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621)
I20240502 01:52:56.418066 2387 raft_consensus.cc:2454] T 1983e0a53ee54b62afa16676bf986332 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 1.
I20240502 01:52:56.418112 2517 raft_consensus.cc:2454] T 1983e0a53ee54b62afa16676bf986332 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 1.
I20240502 01:52:56.418169 2254 raft_consensus.cc:2454] T 1983e0a53ee54b62afa16676bf986332 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 1.
I20240502 01:52:56.418336 2123 raft_consensus.cc:2454] T 1983e0a53ee54b62afa16676bf986332 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 1.
I20240502 01:52:56.418584 2583 leader_election.cc:302] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 446c7f9bf31a4acc9dae8daf43330906, 7ef43ea75ecb46ee905b781c91aed95a, 9a3448a425c041258991dc60dd64ae7d; no voters:
I20240502 01:52:56.418918 3112 raft_consensus.cc:2781] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:56.419332 3112 raft_consensus.cc:686] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [term 1 LEADER]: Becoming Leader. State: Replica: 9a3448a425c041258991dc60dd64ae7d, State: Running, Role: LEADER
I20240502 01:52:56.419796 3112 consensus_queue.cc:237] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } }
I20240502 01:52:56.423224 1841 catalog_manager.cc:5484] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3448a425c041258991dc60dd64ae7d (127.31.161.198). New cstate: current_term: 1 leader_uuid: "9a3448a425c041258991dc60dd64ae7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: HEALTHY } } }
I20240502 01:52:56.444875 2987 raft_consensus.cc:482] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:56.445037 2987 raft_consensus.cc:504] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:56.446048 2987 leader_election.cc:288] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621)
I20240502 01:52:56.446605 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4672bc55a874f159bf10855f0698ba1" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879" is_pre_election: true
I20240502 01:52:56.446921 1990 raft_consensus.cc:2454] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 0.
I20240502 01:52:56.446934 2650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4672bc55a874f159bf10855f0698ba1" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3448a425c041258991dc60dd64ae7d" is_pre_election: true
I20240502 01:52:56.447144 2650 raft_consensus.cc:2454] T e4672bc55a874f159bf10855f0698ba1 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 0.
I20240502 01:52:56.447641 2451 leader_election.cc:302] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 4d99e1fdda704211b3d101f892c57119, 58dae5dc94444d5e92e6907c5b637879, 9a3448a425c041258991dc60dd64ae7d; no voters:
I20240502 01:52:56.447603 2123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4672bc55a874f159bf10855f0698ba1" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea" is_pre_election: true
I20240502 01:52:56.447881 2123 raft_consensus.cc:2454] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 0.
I20240502 01:52:56.447966 2987 raft_consensus.cc:2781] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:56.448098 2987 raft_consensus.cc:482] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:56.448204 2987 raft_consensus.cc:3037] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:56.450855 2987 raft_consensus.cc:504] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:56.451751 2987 leader_election.cc:288] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [CANDIDATE]: Term 1 election: Requested vote from peers 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621)
I20240502 01:52:56.452093 2650 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4672bc55a874f159bf10855f0698ba1" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3448a425c041258991dc60dd64ae7d"
I20240502 01:52:56.452150 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4672bc55a874f159bf10855f0698ba1" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879"
I20240502 01:52:56.452335 1990 raft_consensus.cc:3037] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:56.452328 2650 raft_consensus.cc:3037] T e4672bc55a874f159bf10855f0698ba1 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:56.452550 2123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4672bc55a874f159bf10855f0698ba1" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea"
I20240502 01:52:56.452803 2123 raft_consensus.cc:3037] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:56.454654 2387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4672bc55a874f159bf10855f0698ba1" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a" is_pre_election: true
I20240502 01:52:56.454702 2385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4672bc55a874f159bf10855f0698ba1" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a"
I20240502 01:52:56.454898 2387 raft_consensus.cc:2454] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 0.
I20240502 01:52:56.454977 2650 raft_consensus.cc:2454] T e4672bc55a874f159bf10855f0698ba1 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 1.
I20240502 01:52:56.455659 2123 raft_consensus.cc:2454] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 1.
I20240502 01:52:56.456036 2454 leader_election.cc:302] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 269c9061e1a1435e92f79f0b9486dfea, 4d99e1fdda704211b3d101f892c57119, 9a3448a425c041258991dc60dd64ae7d; no voters: 7ef43ea75ecb46ee905b781c91aed95a
I20240502 01:52:56.456305 2987 raft_consensus.cc:2781] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:56.456445 2987 raft_consensus.cc:686] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [term 1 LEADER]: Becoming Leader. State: Replica: 4d99e1fdda704211b3d101f892c57119, State: Running, Role: LEADER
I20240502 01:52:56.456746 2987 consensus_queue.cc:237] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:56.457645 1990 raft_consensus.cc:2454] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 1.
I20240502 01:52:56.459972 1841 catalog_manager.cc:5484] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d99e1fdda704211b3d101f892c57119 (127.31.161.197). New cstate: current_term: 1 leader_uuid: "4d99e1fdda704211b3d101f892c57119" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: UNKNOWN } } }
W20240502 01:52:56.792181 3302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:56.792495 3302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:56.800017 3302 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 01:52:56.839375 3190 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:56.840386 2987 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:56.855248 2987 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:56.868438 2387 raft_consensus.cc:3037] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:56.868816 3220 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:56.873046 3220 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:56.874255 3190 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:56.878331 3190 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:56.917780 3112 consensus_queue.cc:1035] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:56.955689 3112 consensus_queue.cc:1035] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:56.968930 3342 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:56.969431 3112 consensus_queue.cc:1035] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:57.009979 3112 consensus_queue.cc:1035] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 406.752 ms
time per row: 0.203376 ms
I20240502 01:52:57.411597 1848 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40648:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:52:57.419833 2101 tablet_service.cc:1467] Processing CreateTablet for tablet 27398ad1eb7f4bfab185f8affe1f790d (DEFAULT_TABLE table=rebalance_test_table_4 [id=2cd052d4ae9b43129f79f513560af30d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:57.420321 2101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27398ad1eb7f4bfab185f8affe1f790d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.420378 2234 tablet_service.cc:1467] Processing CreateTablet for tablet 27398ad1eb7f4bfab185f8affe1f790d (DEFAULT_TABLE table=rebalance_test_table_4 [id=2cd052d4ae9b43129f79f513560af30d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:57.420791 2234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27398ad1eb7f4bfab185f8affe1f790d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.421748 2232 tablet_service.cc:1467] Processing CreateTablet for tablet e567c877188549fc965637e89bf437ae (DEFAULT_TABLE table=rebalance_test_table_4 [id=2cd052d4ae9b43129f79f513560af30d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:57.422130 2232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e567c877188549fc965637e89bf437ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.422032 1970 tablet_service.cc:1467] Processing CreateTablet for tablet e567c877188549fc965637e89bf437ae (DEFAULT_TABLE table=rebalance_test_table_4 [id=2cd052d4ae9b43129f79f513560af30d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:57.422439 1970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e567c877188549fc965637e89bf437ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.423553 2366 tablet_service.cc:1467] Processing CreateTablet for tablet e567c877188549fc965637e89bf437ae (DEFAULT_TABLE table=rebalance_test_table_4 [id=2cd052d4ae9b43129f79f513560af30d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:57.423970 2366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e567c877188549fc965637e89bf437ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.424384 1972 tablet_service.cc:1467] Processing CreateTablet for tablet f4a07f0fe1964f958ad7ca4a63d054a6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2cd052d4ae9b43129f79f513560af30d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:57.424778 1972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a07f0fe1964f958ad7ca4a63d054a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.429368 1971 tablet_service.cc:1467] Processing CreateTablet for tablet 27398ad1eb7f4bfab185f8affe1f790d (DEFAULT_TABLE table=rebalance_test_table_4 [id=2cd052d4ae9b43129f79f513560af30d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:57.429769 1971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27398ad1eb7f4bfab185f8affe1f790d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.430335 2498 tablet_service.cc:1467] Processing CreateTablet for tablet f4a07f0fe1964f958ad7ca4a63d054a6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2cd052d4ae9b43129f79f513560af30d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:57.430737 2498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a07f0fe1964f958ad7ca4a63d054a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.431239 2629 tablet_service.cc:1467] Processing CreateTablet for tablet f4a07f0fe1964f958ad7ca4a63d054a6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2cd052d4ae9b43129f79f513560af30d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:57.431804 2629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a07f0fe1964f958ad7ca4a63d054a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.433542 2496 tablet_service.cc:1467] Processing CreateTablet for tablet 27398ad1eb7f4bfab185f8affe1f790d (DEFAULT_TABLE table=rebalance_test_table_4 [id=2cd052d4ae9b43129f79f513560af30d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:57.433904 2496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27398ad1eb7f4bfab185f8affe1f790d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.437521 2365 tablet_service.cc:1467] Processing CreateTablet for tablet 27398ad1eb7f4bfab185f8affe1f790d (DEFAULT_TABLE table=rebalance_test_table_4 [id=2cd052d4ae9b43129f79f513560af30d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:57.437911 2365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27398ad1eb7f4bfab185f8affe1f790d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.438541 2233 tablet_service.cc:1467] Processing CreateTablet for tablet f4a07f0fe1964f958ad7ca4a63d054a6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2cd052d4ae9b43129f79f513560af30d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:57.439003 2233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a07f0fe1964f958ad7ca4a63d054a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.445653 2103 tablet_service.cc:1467] Processing CreateTablet for tablet e567c877188549fc965637e89bf437ae (DEFAULT_TABLE table=rebalance_test_table_4 [id=2cd052d4ae9b43129f79f513560af30d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:57.446178 2103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e567c877188549fc965637e89bf437ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.454003 3394 tablet_bootstrap.cc:492] T e567c877188549fc965637e89bf437ae P 58dae5dc94444d5e92e6907c5b637879: Bootstrap starting.
I20240502 01:52:57.455724 3394 tablet_bootstrap.cc:654] T e567c877188549fc965637e89bf437ae P 58dae5dc94444d5e92e6907c5b637879: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:57.458979 2630 tablet_service.cc:1467] Processing CreateTablet for tablet e567c877188549fc965637e89bf437ae (DEFAULT_TABLE table=rebalance_test_table_4 [id=2cd052d4ae9b43129f79f513560af30d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:57.459522 2630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e567c877188549fc965637e89bf437ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.461396 3395 tablet_bootstrap.cc:492] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119: Bootstrap starting.
I20240502 01:52:57.462371 3394 tablet_bootstrap.cc:492] T e567c877188549fc965637e89bf437ae P 58dae5dc94444d5e92e6907c5b637879: No bootstrap required, opened a new log
I20240502 01:52:57.462518 3394 ts_tablet_manager.cc:1367] T e567c877188549fc965637e89bf437ae P 58dae5dc94444d5e92e6907c5b637879: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 01:52:57.463268 3395 tablet_bootstrap.cc:654] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:57.463292 3394 raft_consensus.cc:348] T e567c877188549fc965637e89bf437ae P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:57.463570 3394 raft_consensus.cc:374] T e567c877188549fc965637e89bf437ae P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:57.463676 3394 raft_consensus.cc:729] T e567c877188549fc965637e89bf437ae P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dae5dc94444d5e92e6907c5b637879, State: Initialized, Role: FOLLOWER
I20240502 01:52:57.463912 3394 consensus_queue.cc:260] T e567c877188549fc965637e89bf437ae P 58dae5dc94444d5e92e6907c5b637879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:57.464051 2365 tablet_service.cc:1467] Processing CreateTablet for tablet f4a07f0fe1964f958ad7ca4a63d054a6 (DEFAULT_TABLE table=rebalance_test_table_4 [id=2cd052d4ae9b43129f79f513560af30d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:52:57.464469 3394 ts_tablet_manager.cc:1398] T e567c877188549fc965637e89bf437ae P 58dae5dc94444d5e92e6907c5b637879: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:52:57.464723 2365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a07f0fe1964f958ad7ca4a63d054a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:52:57.466995 3394 tablet_bootstrap.cc:492] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879: Bootstrap starting.
I20240502 01:52:57.468461 3394 tablet_bootstrap.cc:654] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:57.479868 3394 tablet_bootstrap.cc:492] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879: No bootstrap required, opened a new log
I20240502 01:52:57.480059 3394 ts_tablet_manager.cc:1367] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.003s
I20240502 01:52:57.480780 3394 raft_consensus.cc:348] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:57.481092 3394 raft_consensus.cc:374] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:57.481186 3394 raft_consensus.cc:729] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dae5dc94444d5e92e6907c5b637879, State: Initialized, Role: FOLLOWER
I20240502 01:52:57.481446 3394 consensus_queue.cc:260] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:57.484948 3394 ts_tablet_manager.cc:1398] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:52:57.486179 3394 tablet_bootstrap.cc:492] T 27398ad1eb7f4bfab185f8affe1f790d P 58dae5dc94444d5e92e6907c5b637879: Bootstrap starting.
I20240502 01:52:57.489910 3394 tablet_bootstrap.cc:654] T 27398ad1eb7f4bfab185f8affe1f790d P 58dae5dc94444d5e92e6907c5b637879: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:57.491385 3400 tablet_bootstrap.cc:492] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d: Bootstrap starting.
I20240502 01:52:57.492722 3397 tablet_bootstrap.cc:492] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea: Bootstrap starting.
I20240502 01:52:57.493400 3401 tablet_bootstrap.cc:492] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a: Bootstrap starting.
I20240502 01:52:57.494550 3397 tablet_bootstrap.cc:654] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:57.495383 3400 tablet_bootstrap.cc:654] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:57.496652 3394 tablet_bootstrap.cc:492] T 27398ad1eb7f4bfab185f8affe1f790d P 58dae5dc94444d5e92e6907c5b637879: No bootstrap required, opened a new log
I20240502 01:52:57.496805 3394 ts_tablet_manager.cc:1367] T 27398ad1eb7f4bfab185f8affe1f790d P 58dae5dc94444d5e92e6907c5b637879: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 01:52:57.497143 3401 tablet_bootstrap.cc:654] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:57.497555 3394 raft_consensus.cc:348] T 27398ad1eb7f4bfab185f8affe1f790d P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:57.497802 3394 raft_consensus.cc:374] T 27398ad1eb7f4bfab185f8affe1f790d P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:57.497895 3394 raft_consensus.cc:729] T 27398ad1eb7f4bfab185f8affe1f790d P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dae5dc94444d5e92e6907c5b637879, State: Initialized, Role: FOLLOWER
I20240502 01:52:57.499855 3400 tablet_bootstrap.cc:492] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d: No bootstrap required, opened a new log
I20240502 01:52:57.499868 3394 consensus_queue.cc:260] T 27398ad1eb7f4bfab185f8affe1f790d P 58dae5dc94444d5e92e6907c5b637879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:57.500043 3400 ts_tablet_manager.cc:1367] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 01:52:57.501003 3400 raft_consensus.cc:348] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:57.501325 3400 raft_consensus.cc:374] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:57.501468 3400 raft_consensus.cc:729] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3448a425c041258991dc60dd64ae7d, State: Initialized, Role: FOLLOWER
I20240502 01:52:57.501758 3400 consensus_queue.cc:260] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:57.502578 3400 ts_tablet_manager.cc:1398] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:52:57.502907 3400 tablet_bootstrap.cc:492] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d: Bootstrap starting.
I20240502 01:52:57.503319 3394 ts_tablet_manager.cc:1398] T 27398ad1eb7f4bfab185f8affe1f790d P 58dae5dc94444d5e92e6907c5b637879: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:52:57.504477 3400 tablet_bootstrap.cc:654] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:57.504994 3393 tablet_bootstrap.cc:492] T 27398ad1eb7f4bfab185f8affe1f790d P 446c7f9bf31a4acc9dae8daf43330906: Bootstrap starting.
I20240502 01:52:57.506516 3395 tablet_bootstrap.cc:492] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119: No bootstrap required, opened a new log
I20240502 01:52:57.506871 3401 tablet_bootstrap.cc:492] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a: No bootstrap required, opened a new log
I20240502 01:52:57.506841 3395 ts_tablet_manager.cc:1367] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119: Time spent bootstrapping tablet: real 0.046s user 0.020s sys 0.024s
I20240502 01:52:57.507052 3401 ts_tablet_manager.cc:1367] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20240502 01:52:57.507870 3395 raft_consensus.cc:348] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:57.507846 3401 raft_consensus.cc:348] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:57.508141 3395 raft_consensus.cc:374] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:57.508160 3401 raft_consensus.cc:374] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:57.508263 3397 tablet_bootstrap.cc:492] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea: No bootstrap required, opened a new log
I20240502 01:52:57.508265 3395 raft_consensus.cc:729] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d99e1fdda704211b3d101f892c57119, State: Initialized, Role: FOLLOWER
I20240502 01:52:57.508283 3401 raft_consensus.cc:729] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43ea75ecb46ee905b781c91aed95a, State: Initialized, Role: FOLLOWER
I20240502 01:52:57.508424 3397 ts_tablet_manager.cc:1367] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.003s
I20240502 01:52:57.508553 3401 consensus_queue.cc:260] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:57.508572 3395 consensus_queue.cc:260] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:57.509267 3395 ts_tablet_manager.cc:1398] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:52:57.509209 3397 raft_consensus.cc:348] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:57.509481 3397 raft_consensus.cc:374] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:57.509550 3395 tablet_bootstrap.cc:492] T 27398ad1eb7f4bfab185f8affe1f790d P 4d99e1fdda704211b3d101f892c57119: Bootstrap starting.
I20240502 01:52:57.509596 3397 raft_consensus.cc:729] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269c9061e1a1435e92f79f0b9486dfea, State: Initialized, Role: FOLLOWER
I20240502 01:52:57.510468 3401 ts_tablet_manager.cc:1398] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:52:57.509876 3397 consensus_queue.cc:260] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:57.510941 3401 tablet_bootstrap.cc:492] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a: Bootstrap starting.
I20240502 01:52:57.511080 3395 tablet_bootstrap.cc:654] T 27398ad1eb7f4bfab185f8affe1f790d P 4d99e1fdda704211b3d101f892c57119: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:57.511152 3397 ts_tablet_manager.cc:1398] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:52:57.511451 3397 tablet_bootstrap.cc:492] T 27398ad1eb7f4bfab185f8affe1f790d P 269c9061e1a1435e92f79f0b9486dfea: Bootstrap starting.
I20240502 01:52:57.512890 3397 tablet_bootstrap.cc:654] T 27398ad1eb7f4bfab185f8affe1f790d P 269c9061e1a1435e92f79f0b9486dfea: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:57.514629 3401 tablet_bootstrap.cc:654] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:57.515681 3393 tablet_bootstrap.cc:654] T 27398ad1eb7f4bfab185f8affe1f790d P 446c7f9bf31a4acc9dae8daf43330906: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:57.521991 3400 tablet_bootstrap.cc:492] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d: No bootstrap required, opened a new log
I20240502 01:52:57.522781 3397 tablet_bootstrap.cc:492] T 27398ad1eb7f4bfab185f8affe1f790d P 269c9061e1a1435e92f79f0b9486dfea: No bootstrap required, opened a new log
I20240502 01:52:57.522948 3397 ts_tablet_manager.cc:1367] T 27398ad1eb7f4bfab185f8affe1f790d P 269c9061e1a1435e92f79f0b9486dfea: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.003s
I20240502 01:52:57.523061 3401 tablet_bootstrap.cc:492] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a: No bootstrap required, opened a new log
I20240502 01:52:57.523049 3400 ts_tablet_manager.cc:1367] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240502 01:52:57.523072 3395 tablet_bootstrap.cc:492] T 27398ad1eb7f4bfab185f8affe1f790d P 4d99e1fdda704211b3d101f892c57119: No bootstrap required, opened a new log
I20240502 01:52:57.522791 3393 tablet_bootstrap.cc:492] T 27398ad1eb7f4bfab185f8affe1f790d P 446c7f9bf31a4acc9dae8daf43330906: No bootstrap required, opened a new log
I20240502 01:52:57.523211 3395 ts_tablet_manager.cc:1367] T 27398ad1eb7f4bfab185f8affe1f790d P 4d99e1fdda704211b3d101f892c57119: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.006s
I20240502 01:52:57.523216 3401 ts_tablet_manager.cc:1367] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s
I20240502 01:52:57.523797 3397 raft_consensus.cc:348] T 27398ad1eb7f4bfab185f8affe1f790d P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:57.523867 3395 raft_consensus.cc:348] T 27398ad1eb7f4bfab185f8affe1f790d P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:57.523861 3400 raft_consensus.cc:348] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:57.524068 3397 raft_consensus.cc:374] T 27398ad1eb7f4bfab185f8affe1f790d P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:57.524108 3395 raft_consensus.cc:374] T 27398ad1eb7f4bfab185f8affe1f790d P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:57.524109 3400 raft_consensus.cc:374] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:57.523977 3401 raft_consensus.cc:348] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:57.524135 3397 raft_consensus.cc:729] T 27398ad1eb7f4bfab185f8affe1f790d P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269c9061e1a1435e92f79f0b9486dfea, State: Initialized, Role: FOLLOWER
I20240502 01:52:57.524175 3395 raft_consensus.cc:729] T 27398ad1eb7f4bfab185f8affe1f790d P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d99e1fdda704211b3d101f892c57119, State: Initialized, Role: FOLLOWER
I20240502 01:52:57.524175 3400 raft_consensus.cc:729] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3448a425c041258991dc60dd64ae7d, State: Initialized, Role: FOLLOWER
I20240502 01:52:57.524212 3401 raft_consensus.cc:374] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:57.524324 3401 raft_consensus.cc:729] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43ea75ecb46ee905b781c91aed95a, State: Initialized, Role: FOLLOWER
I20240502 01:52:57.524435 3393 ts_tablet_manager.cc:1367] T 27398ad1eb7f4bfab185f8affe1f790d P 446c7f9bf31a4acc9dae8daf43330906: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240502 01:52:57.524396 3397 consensus_queue.cc:260] T 27398ad1eb7f4bfab185f8affe1f790d P 269c9061e1a1435e92f79f0b9486dfea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:57.524405 3395 consensus_queue.cc:260] T 27398ad1eb7f4bfab185f8affe1f790d P 4d99e1fdda704211b3d101f892c57119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:57.524580 3401 consensus_queue.cc:260] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:57.525146 3401 ts_tablet_manager.cc:1398] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 01:52:57.525724 3395 ts_tablet_manager.cc:1398] T 27398ad1eb7f4bfab185f8affe1f790d P 4d99e1fdda704211b3d101f892c57119: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:52:57.525806 3393 raft_consensus.cc:348] T 27398ad1eb7f4bfab185f8affe1f790d P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:57.526160 3400 consensus_queue.cc:260] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:57.526434 3393 raft_consensus.cc:374] T 27398ad1eb7f4bfab185f8affe1f790d P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:57.526540 3393 raft_consensus.cc:729] T 27398ad1eb7f4bfab185f8affe1f790d P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446c7f9bf31a4acc9dae8daf43330906, State: Initialized, Role: FOLLOWER
I20240502 01:52:57.526842 3341 raft_consensus.cc:482] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:57.526767 3393 consensus_queue.cc:260] T 27398ad1eb7f4bfab185f8affe1f790d P 446c7f9bf31a4acc9dae8daf43330906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:57.526978 3341 raft_consensus.cc:504] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:57.527967 3400 ts_tablet_manager.cc:1398] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 01:52:57.528030 3341 leader_election.cc:288] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987)
I20240502 01:52:57.528368 2254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27398ad1eb7f4bfab185f8affe1f790d" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906" is_pre_election: true
I20240502 01:52:57.528573 2254 raft_consensus.cc:2454] T 27398ad1eb7f4bfab185f8affe1f790d P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 0.
I20240502 01:52:57.528941 2517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27398ad1eb7f4bfab185f8affe1f790d" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119" is_pre_election: true
I20240502 01:52:57.528941 2124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27398ad1eb7f4bfab185f8affe1f790d" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea" is_pre_election: true
I20240502 01:52:57.529163 2517 raft_consensus.cc:2454] T 27398ad1eb7f4bfab185f8affe1f790d P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 0.
I20240502 01:52:57.529163 2124 raft_consensus.cc:2454] T 27398ad1eb7f4bfab185f8affe1f790d P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 0.
I20240502 01:52:57.529068 3393 ts_tablet_manager.cc:1398] T 27398ad1eb7f4bfab185f8affe1f790d P 446c7f9bf31a4acc9dae8daf43330906: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:52:57.529460 3393 tablet_bootstrap.cc:492] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906: Bootstrap starting.
I20240502 01:52:57.529557 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27398ad1eb7f4bfab185f8affe1f790d" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879" is_pre_election: true
I20240502 01:52:57.529803 1990 raft_consensus.cc:2454] T 27398ad1eb7f4bfab185f8affe1f790d P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 0.
I20240502 01:52:57.529798 2322 leader_election.cc:302] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 269c9061e1a1435e92f79f0b9486dfea, 4d99e1fdda704211b3d101f892c57119, 7ef43ea75ecb46ee905b781c91aed95a; no voters:
I20240502 01:52:57.530076 3341 raft_consensus.cc:2781] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:57.530189 3341 raft_consensus.cc:482] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:57.530265 3341 raft_consensus.cc:3037] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:57.531046 3393 tablet_bootstrap.cc:654] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:57.532843 3401 tablet_bootstrap.cc:492] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a: Bootstrap starting.
I20240502 01:52:57.533000 3341 raft_consensus.cc:504] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:57.533923 3341 leader_election.cc:288] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 election: Requested vote from peers 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987)
I20240502 01:52:57.534623 2124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27398ad1eb7f4bfab185f8affe1f790d" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea"
I20240502 01:52:57.534822 2124 raft_consensus.cc:3037] T 27398ad1eb7f4bfab185f8affe1f790d P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:57.536679 3397 ts_tablet_manager.cc:1398] T 27398ad1eb7f4bfab185f8affe1f790d P 269c9061e1a1435e92f79f0b9486dfea: Time spent starting tablet: real 0.014s user 0.002s sys 0.001s
I20240502 01:52:57.537431 2517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27398ad1eb7f4bfab185f8affe1f790d" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119"
I20240502 01:52:57.537487 2124 raft_consensus.cc:2454] T 27398ad1eb7f4bfab185f8affe1f790d P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 1.
I20240502 01:52:57.537626 2517 raft_consensus.cc:3037] T 27398ad1eb7f4bfab185f8affe1f790d P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:57.537551 2254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27398ad1eb7f4bfab185f8affe1f790d" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906"
I20240502 01:52:57.537771 2254 raft_consensus.cc:3037] T 27398ad1eb7f4bfab185f8affe1f790d P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:57.539453 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27398ad1eb7f4bfab185f8affe1f790d" candidate_uuid: "7ef43ea75ecb46ee905b781c91aed95a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879"
I20240502 01:52:57.539683 1990 raft_consensus.cc:3037] T 27398ad1eb7f4bfab185f8affe1f790d P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:57.540247 2517 raft_consensus.cc:2454] T 27398ad1eb7f4bfab185f8affe1f790d P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 1.
I20240502 01:52:57.540601 2254 raft_consensus.cc:2454] T 27398ad1eb7f4bfab185f8affe1f790d P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 1.
I20240502 01:52:57.540669 2319 leader_election.cc:302] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 269c9061e1a1435e92f79f0b9486dfea, 4d99e1fdda704211b3d101f892c57119, 7ef43ea75ecb46ee905b781c91aed95a; no voters:
I20240502 01:52:57.540994 3341 raft_consensus.cc:2781] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:57.541266 3341 raft_consensus.cc:686] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Becoming Leader. State: Replica: 7ef43ea75ecb46ee905b781c91aed95a, State: Running, Role: LEADER
I20240502 01:52:57.541545 3341 consensus_queue.cc:237] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } }
I20240502 01:52:57.542495 1990 raft_consensus.cc:2454] T 27398ad1eb7f4bfab185f8affe1f790d P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ef43ea75ecb46ee905b781c91aed95a in term 1.
I20240502 01:52:57.543488 3401 tablet_bootstrap.cc:654] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:57.545470 3401 tablet_bootstrap.cc:492] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a: No bootstrap required, opened a new log
I20240502 01:52:57.545600 3401 ts_tablet_manager.cc:1367] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240502 01:52:57.545504 1841 catalog_manager.cc:5484] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196). New cstate: current_term: 1 leader_uuid: "7ef43ea75ecb46ee905b781c91aed95a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:57.546181 3401 raft_consensus.cc:348] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:57.546458 3393 tablet_bootstrap.cc:492] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906: No bootstrap required, opened a new log
I20240502 01:52:57.546468 3401 raft_consensus.cc:374] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:57.546552 3401 raft_consensus.cc:729] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ef43ea75ecb46ee905b781c91aed95a, State: Initialized, Role: FOLLOWER
I20240502 01:52:57.546618 3393 ts_tablet_manager.cc:1367] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
I20240502 01:52:57.546773 3401 consensus_queue.cc:260] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:57.547345 3401 ts_tablet_manager.cc:1398] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:52:57.547375 3393 raft_consensus.cc:348] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:57.547616 3393 raft_consensus.cc:374] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:57.547698 3393 raft_consensus.cc:729] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446c7f9bf31a4acc9dae8daf43330906, State: Initialized, Role: FOLLOWER
I20240502 01:52:57.547953 3393 consensus_queue.cc:260] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:57.548511 3393 ts_tablet_manager.cc:1398] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:52:57.548780 3393 tablet_bootstrap.cc:492] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906: Bootstrap starting.
I20240502 01:52:57.550159 3393 tablet_bootstrap.cc:654] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906: Neither blocks nor log segments found. Creating new log.
I20240502 01:52:57.552503 3393 tablet_bootstrap.cc:492] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906: No bootstrap required, opened a new log
I20240502 01:52:57.552628 3393 ts_tablet_manager.cc:1367] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:52:57.553196 3393 raft_consensus.cc:348] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:57.553406 3393 raft_consensus.cc:374] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:52:57.553498 3393 raft_consensus.cc:729] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446c7f9bf31a4acc9dae8daf43330906, State: Initialized, Role: FOLLOWER
I20240502 01:52:57.553710 3393 consensus_queue.cc:260] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:57.554246 3393 ts_tablet_manager.cc:1398] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:52:57.560166 3329 raft_consensus.cc:482] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:57.560303 3329 raft_consensus.cc:504] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:57.561097 3329 leader_election.cc:288] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225)
I20240502 01:52:57.561434 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a07f0fe1964f958ad7ca4a63d054a6" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879" is_pre_election: true
I20240502 01:52:57.561429 2517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a07f0fe1964f958ad7ca4a63d054a6" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119" is_pre_election: true
I20240502 01:52:57.561522 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a07f0fe1964f958ad7ca4a63d054a6" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3448a425c041258991dc60dd64ae7d" is_pre_election: true
I20240502 01:52:57.561646 1990 raft_consensus.cc:2454] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 446c7f9bf31a4acc9dae8daf43330906 in term 0.
I20240502 01:52:57.561679 2517 raft_consensus.cc:2454] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 446c7f9bf31a4acc9dae8daf43330906 in term 0.
I20240502 01:52:57.561720 2648 raft_consensus.cc:2454] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 446c7f9bf31a4acc9dae8daf43330906 in term 0.
I20240502 01:52:57.561746 2387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a07f0fe1964f958ad7ca4a63d054a6" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a" is_pre_election: true
I20240502 01:52:57.561956 2387 raft_consensus.cc:2454] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 446c7f9bf31a4acc9dae8daf43330906 in term 0.
I20240502 01:52:57.562129 2187 leader_election.cc:302] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 446c7f9bf31a4acc9dae8daf43330906, 58dae5dc94444d5e92e6907c5b637879, 9a3448a425c041258991dc60dd64ae7d; no voters:
I20240502 01:52:57.562397 3329 raft_consensus.cc:2781] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:57.562500 3329 raft_consensus.cc:482] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:57.562582 3329 raft_consensus.cc:3037] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:57.564155 3349 raft_consensus.cc:482] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:52:57.564325 3349 raft_consensus.cc:504] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:57.564582 3329 raft_consensus.cc:504] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:57.565490 3349 leader_election.cc:288] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621)
I20240502 01:52:57.565857 1990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e567c877188549fc965637e89bf437ae" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879" is_pre_election: true
I20240502 01:52:57.565953 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a07f0fe1964f958ad7ca4a63d054a6" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879"
I20240502 01:52:57.565943 2517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a07f0fe1964f958ad7ca4a63d054a6" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119"
I20240502 01:52:57.566051 1990 raft_consensus.cc:2454] T e567c877188549fc965637e89bf437ae P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 0.
I20240502 01:52:57.566098 1991 raft_consensus.cc:3037] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:57.566066 2648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a07f0fe1964f958ad7ca4a63d054a6" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3448a425c041258991dc60dd64ae7d"
I20240502 01:52:57.566175 2517 raft_consensus.cc:3037] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:57.566219 2648 raft_consensus.cc:3037] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:57.566370 3329 leader_election.cc:288] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [CANDIDATE]: Term 1 election: Requested vote from peers 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225)
I20240502 01:52:57.567513 2254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e567c877188549fc965637e89bf437ae" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906" is_pre_election: true
I20240502 01:52:57.567718 2254 raft_consensus.cc:2454] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 0.
I20240502 01:52:57.568198 2387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e567c877188549fc965637e89bf437ae" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a" is_pre_election: true
I20240502 01:52:57.568405 2387 raft_consensus.cc:2454] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 0.
I20240502 01:52:57.568640 1991 raft_consensus.cc:2454] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446c7f9bf31a4acc9dae8daf43330906 in term 1.
I20240502 01:52:57.568918 2648 raft_consensus.cc:2454] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446c7f9bf31a4acc9dae8daf43330906 in term 1.
I20240502 01:52:57.568907 2517 raft_consensus.cc:2454] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446c7f9bf31a4acc9dae8daf43330906 in term 1.
I20240502 01:52:57.569051 2124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e567c877188549fc965637e89bf437ae" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea" is_pre_election: true
I20240502 01:52:57.569291 2585 leader_election.cc:302] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 58dae5dc94444d5e92e6907c5b637879, 7ef43ea75ecb46ee905b781c91aed95a, 9a3448a425c041258991dc60dd64ae7d; no voters:
I20240502 01:52:57.569350 2124 raft_consensus.cc:2454] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 0.
I20240502 01:52:57.569555 3349 raft_consensus.cc:2781] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:52:57.569658 3349 raft_consensus.cc:482] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:52:57.569746 3349 raft_consensus.cc:3037] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:57.569947 2187 leader_election.cc:302] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 446c7f9bf31a4acc9dae8daf43330906, 58dae5dc94444d5e92e6907c5b637879, 9a3448a425c041258991dc60dd64ae7d; no voters:
I20240502 01:52:57.570250 2387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a07f0fe1964f958ad7ca4a63d054a6" candidate_uuid: "446c7f9bf31a4acc9dae8daf43330906" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a"
I20240502 01:52:57.570319 3329 raft_consensus.cc:2781] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:57.570438 2387 raft_consensus.cc:3037] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:57.571704 3329 raft_consensus.cc:686] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEADER]: Becoming Leader. State: Replica: 446c7f9bf31a4acc9dae8daf43330906, State: Running, Role: LEADER
I20240502 01:52:57.572238 3349 raft_consensus.cc:504] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:57.572269 3329 consensus_queue.cc:237] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } }
I20240502 01:52:57.573413 2387 raft_consensus.cc:2454] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446c7f9bf31a4acc9dae8daf43330906 in term 1.
I20240502 01:52:57.573438 2254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e567c877188549fc965637e89bf437ae" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446c7f9bf31a4acc9dae8daf43330906"
I20240502 01:52:57.573392 2385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e567c877188549fc965637e89bf437ae" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a"
I20240502 01:52:57.573623 2254 raft_consensus.cc:3037] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:57.573632 2385 raft_consensus.cc:3037] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:57.576292 2385 raft_consensus.cc:2454] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 1.
I20240502 01:52:57.576426 2254 raft_consensus.cc:2454] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 1.
I20240502 01:52:57.576750 1991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e567c877188549fc965637e89bf437ae" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dae5dc94444d5e92e6907c5b637879"
I20240502 01:52:57.576905 3349 leader_election.cc:288] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [CANDIDATE]: Term 1 election: Requested vote from peers 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621)
I20240502 01:52:57.576918 1991 raft_consensus.cc:3037] T e567c877188549fc965637e89bf437ae P 58dae5dc94444d5e92e6907c5b637879 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:57.577371 2583 leader_election.cc:302] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 446c7f9bf31a4acc9dae8daf43330906, 7ef43ea75ecb46ee905b781c91aed95a, 9a3448a425c041258991dc60dd64ae7d; no voters:
I20240502 01:52:57.577440 2124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e567c877188549fc965637e89bf437ae" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269c9061e1a1435e92f79f0b9486dfea"
I20240502 01:52:57.577663 2124 raft_consensus.cc:3037] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:52:57.577898 3348 raft_consensus.cc:2781] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:52:57.578016 3348 raft_consensus.cc:686] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [term 1 LEADER]: Becoming Leader. State: Replica: 9a3448a425c041258991dc60dd64ae7d, State: Running, Role: LEADER
I20240502 01:52:57.578317 3348 consensus_queue.cc:237] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } }
I20240502 01:52:57.579471 1991 raft_consensus.cc:2454] T e567c877188549fc965637e89bf437ae P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 1.
I20240502 01:52:57.579991 2124 raft_consensus.cc:2454] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3448a425c041258991dc60dd64ae7d in term 1.
I20240502 01:52:57.581939 1841 catalog_manager.cc:5484] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 reported cstate change: term changed from 0 to 1, leader changed from <none> to 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195). New cstate: current_term: 1 leader_uuid: "446c7f9bf31a4acc9dae8daf43330906" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: UNKNOWN } } }
I20240502 01:52:57.586324 1841 catalog_manager.cc:5484] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3448a425c041258991dc60dd64ae7d (127.31.161.198). New cstate: current_term: 1 leader_uuid: "9a3448a425c041258991dc60dd64ae7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: UNKNOWN } } }
W20240502 01:52:57.930137 3419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:57.930413 3419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:57.937542 3419 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:52:57.962424 3341 consensus_queue.cc:1035] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240502 01:52:58.065585 3348 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:58.072701 3348 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:58.117921 3341 consensus_queue.cc:1035] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:58.127583 3329 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:58.144470 3329 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:58.158810 3329 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:58.159317 2385 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:58.161257 3341 consensus_queue.cc:1035] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:58.161974 3467 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:58.169281 3341 consensus_queue.cc:1035] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:58.239147 3485 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:58.270418 2253 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:52:58.271157 3467 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:52:58.365219 3471 mvcc.cc:205] Tried to move back new op lower bound from 7023062132110786560 to 7023062128816361472. Current Snapshot: MvccSnapshot[applied={T|T < 7023062131629957120}]
I20240502 01:52:58.389668 3462 mvcc.cc:205] Tried to move back new op lower bound from 7023062132169433088 to 7023062128940060672. Current Snapshot: MvccSnapshot[applied={T|T < 7023062131737174016}]
I20240502 01:52:58.394630 3460 mvcc.cc:205] Tried to move back new op lower bound from 7023062131222523904 to 7023062128964603904. Current Snapshot: MvccSnapshot[applied={T|T < 7023062131222523904}]
INSERT report
rows total: 2000
time total: 518.747 ms
time per row: 0.259374 ms
I20240502 01:52:58.604097 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.199:46381
--local_ip_for_outbound_sockets=127.31.161.199
--tserver_master_addrs=127.31.161.254:45525
--webserver_port=44149
--webserver_interface=127.31.161.199
--builtin_ntp_servers=127.31.161.212:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:52:58.928761 3522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:52:58.929018 3522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:52:58.929137 3522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:52:58.936336 3522 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:52:58.936441 3522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:52:58.936554 3522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.199
I20240502 01:52:58.945488 3522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.199:46381
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.31.161.199
--webserver_port=44149
--tserver_master_addrs=127.31.161.254:45525
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.199
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:52:58.946686 3522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:52:58.948617 3522 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:52:58.960878 3527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:58.961580 3528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:52:58.962064 3530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:52:58.963378 3522 server_base.cc:1034] running on GCE node
I20240502 01:52:58.964112 3522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:52:58.965019 3522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:52:58.966289 3522 hybrid_clock.cc:648] HybridClock initialized: now 1714614778966232 us; error 40 us; skew 500 ppm
I20240502 01:52:58.966619 3522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:52:58.968824 3522 webserver.cc:460] Webserver started at http://127.31.161.199:44149/ using document root <none> and password file <none>
I20240502 01:52:58.969659 3522 fs_manager.cc:359] Metadata directory not provided
I20240502 01:52:58.969782 3522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:52:58.975064 3522 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 01:52:58.977762 3536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:58.979027 3522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:52:58.979168 3522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "163649b5fbb140f18dccdf75f608b0f9"
format_stamp: "Formatted at 2024-05-02 01:52:47 on dist-test-slave-06l8"
I20240502 01:52:58.979939 3522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:52:58.997750 3522 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:52:58.998473 3522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:52:58.998729 3522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:52:58.999575 3522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:52:59.001222 3522 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:52:59.001320 3522 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:59.001446 3522 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:52:59.001497 3522 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:52:59.037734 3522 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.199:46381
I20240502 01:52:59.037706 3650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.199:46381 every 8 connection(s)
I20240502 01:52:59.039438 3522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:52:59.039950 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 3522
I20240502 01:52:59.040241 32391 test_util.cc:268] Using random seed: -590032423
I20240502 01:52:59.052970 3651 heartbeater.cc:347] Connected to a master server at 127.31.161.254:45525
I20240502 01:52:59.053252 3651 heartbeater.cc:464] Registering TS with master...
I20240502 01:52:59.053979 3651 heartbeater.cc:511] Master 127.31.161.254:45525 requested a full tablet report, sending...
I20240502 01:52:59.055467 1842 ts_manager.cc:194] Re-registered known tserver with Master: 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381)
I20240502 01:52:59.056293 1842 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.199:53875
I20240502 01:52:59.063620 32391 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240502 01:52:59.064028 32391 test_util.cc:268] Using random seed: -590008636
I20240502 01:52:59.078415 32391 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240502 01:52:59.081362 32391 test_util.cc:268] Using random seed: -589991316
W20240502 01:52:59.221485 2189 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 48555.4us
I20240502 01:52:59.226588 32391 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240502 01:52:59.240552 32391 test_util.cc:268] Using random seed: -589832120
I20240502 01:52:59.356367 32391 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240502 01:52:59.364830 32391 test_util.cc:268] Using random seed: -589707845
I20240502 01:52:59.524561 32391 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
W20240502 01:52:59.553587 1925 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.198:37585, user_credentials={real_user=slave}} blocked reactor thread for 68528.5us
I20240502 01:53:00.058916 3651 heartbeater.cc:503] Master 127.31.161.254:45525 was elected leader, sending a full tablet report...
W20240502 01:53:00.518378 2454 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 56468.8us
W20240502 01:53:01.031761 2188 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.196:39225, user_credentials={real_user=slave}} blocked reactor thread for 48539.3us
W20240502 01:53:01.211150 2586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 47846us
W20240502 01:53:01.432605 2322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 44736.2us
W20240502 01:53:02.374747 2319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.197:38987, user_credentials={real_user=slave}} blocked reactor thread for 47178.3us
W20240502 01:53:02.488179 2319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.198:37585, user_credentials={real_user=slave}} blocked reactor thread for 45638us
I20240502 01:53:02.905620 2103 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:02.916193 2234 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:02.956691 3585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:03.099642 2496 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:03.184255 2630 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:03.210914 1971 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:53:03.245354 2322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 44362.1us
I20240502 01:53:03.323993 2366 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:53:03.767706 2321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 46927.5us
W20240502 01:53:03.767964 2322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 45682.1us
W20240502 01:53:03.771595 2319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.197:38987, user_credentials={real_user=slave}} blocked reactor thread for 55762.9us
W20240502 01:53:04.333653 2319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.195:39353, user_credentials={real_user=slave}} blocked reactor thread for 166000us
W20240502 01:53:04.395139 2319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.195:39353, user_credentials={real_user=slave}} blocked reactor thread for 54996.8us
I20240502 01:53:05.049774 2103 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:05.050760 3585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:53:05.114403 1925 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.195:39353, user_credentials={real_user=slave}} blocked reactor thread for 57181.2us
I20240502 01:53:05.227283 1971 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:05.346659 2496 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:05.390676 2366 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:05.409458 2234 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:05.456902 2630 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:53:06.279687 2187 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.198:37585, user_credentials={real_user=slave}} blocked reactor thread for 93866.5us
I20240502 01:53:06.316742 2649 consensus_queue.cc:237] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 1.73, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:06.321609 2120 raft_consensus.cc:1261] T 1983e0a53ee54b62afa16676bf986332 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240502 01:53:06.322746 3933 consensus_queue.cc:1035] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240502 01:53:06.324153 2254 raft_consensus.cc:1261] T 1983e0a53ee54b62afa16676bf986332 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240502 01:53:06.325223 3721 consensus_queue.cc:1035] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240502 01:53:06.327769 2517 raft_consensus.cc:1261] T 1983e0a53ee54b62afa16676bf986332 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240502 01:53:06.333495 3721 raft_consensus.cc:2932] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [term 1 LEADER]: Committing config change with OpId 1.74: config changed from index -1 to 74, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:06.339646 2124 raft_consensus.cc:2932] T 1983e0a53ee54b62afa16676bf986332 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index -1 to 74, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:06.342068 3721 consensus_queue.cc:1035] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240502 01:53:06.342430 1842 catalog_manager.cc:5484] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d reported cstate change: config changed from index -1 to 74, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New cstate: current_term: 1 leader_uuid: "9a3448a425c041258991dc60dd64ae7d" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:06.343989 2517 raft_consensus.cc:2932] T 1983e0a53ee54b62afa16676bf986332 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index -1 to 74, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:06.344700 2254 raft_consensus.cc:2932] T 1983e0a53ee54b62afa16676bf986332 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index -1 to 74, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
W20240502 01:53:06.394187 2585 consensus_peers.cc:485] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d -> Peer 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381): Couldn't send request to peer 163649b5fbb140f18dccdf75f608b0f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1983e0a53ee54b62afa16676bf986332. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:06.398885 2384 raft_consensus.cc:1261] T 1983e0a53ee54b62afa16676bf986332 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240502 01:53:06.402372 3721 consensus_queue.cc:1035] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240502 01:53:06.408227 2385 raft_consensus.cc:2932] T 1983e0a53ee54b62afa16676bf986332 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index -1 to 74, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:06.761152 2644 consensus_queue.cc:237] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 1.50, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:06.763825 2122 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 50. Preceding OpId from leader: term: 1 index: 51. (index mismatch)
I20240502 01:53:06.765034 3721 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
W20240502 01:53:06.765417 2585 consensus_peers.cc:485] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d -> Peer 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381): Couldn't send request to peer 163649b5fbb140f18dccdf75f608b0f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e567c877188549fc965637e89bf437ae. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:06.765734 1990 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 50. Preceding OpId from leader: term: 1 index: 51. (index mismatch)
I20240502 01:53:06.768416 4038 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20240502 01:53:06.772759 4038 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [term 1 LEADER]: Committing config change with OpId 1.51: config changed from index -1 to 51, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:06.775007 1990 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.51: config changed from index -1 to 51, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:06.780256 2122 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.51: config changed from index -1 to 51, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:06.783298 2384 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 50. Preceding OpId from leader: term: 1 index: 51. (index mismatch)
I20240502 01:53:06.786866 2251 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 50. Preceding OpId from leader: term: 1 index: 51. (index mismatch)
I20240502 01:53:06.787299 3721 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20240502 01:53:06.791291 2384 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.51: config changed from index -1 to 51, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
W20240502 01:53:06.794329 2319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.195:39353, user_credentials={real_user=slave}} blocked reactor thread for 66320.9us
I20240502 01:53:06.816205 4038 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 49, Time since last communication: 0.000s
I20240502 01:53:06.827011 2253 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.51: config changed from index -1 to 51, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:06.829663 4044 ts_tablet_manager.cc:902] T 1983e0a53ee54b62afa16676bf986332 P 163649b5fbb140f18dccdf75f608b0f9: Initiating tablet copy from peer 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585)
I20240502 01:53:06.858052 1847 catalog_manager.cc:5484] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea reported cstate change: config changed from index -1 to 51, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New cstate: current_term: 1 leader_uuid: "9a3448a425c041258991dc60dd64ae7d" committed_config { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:07.032359 2516 consensus_queue.cc:237] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.78, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: true } }
I20240502 01:53:07.039358 2644 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240502 01:53:07.042837 1990 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240502 01:53:07.043510 2387 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240502 01:53:07.045450 3688 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
W20240502 01:53:07.046591 2451 consensus_peers.cc:485] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 -> Peer 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353): Couldn't send request to peer 446c7f9bf31a4acc9dae8daf43330906. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4672bc55a874f159bf10855f0698ba1. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:07.059602 3941 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240502 01:53:07.065768 3941 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240502 01:53:07.069579 3941 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [term 1 LEADER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: true } } }
I20240502 01:53:07.070047 1990 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: true } } }
I20240502 01:53:07.076733 2644 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: true } } }
I20240502 01:53:07.082996 2383 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: true } } }
I20240502 01:53:07.085613 1844 catalog_manager.cc:5484] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 reported cstate change: config changed from index -1 to 79, NON_VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) added. New cstate: current_term: 1 leader_uuid: "4d99e1fdda704211b3d101f892c57119" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:07.090193 2124 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240502 01:53:07.093976 4044 tablet_copy_client.cc:308] T 1983e0a53ee54b62afa16676bf986332 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.198:37585
I20240502 01:53:07.098484 3724 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 77, Time since last communication: 0.000s
I20240502 01:53:07.101370 2119 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: true } } }
I20240502 01:53:07.189685 2670 tablet_copy_service.cc:140] P 9a3448a425c041258991dc60dd64ae7d: Received BeginTabletCopySession request for tablet 1983e0a53ee54b62afa16676bf986332 from peer 163649b5fbb140f18dccdf75f608b0f9 ({username='slave'} at 127.31.161.199:56747)
I20240502 01:53:07.189980 2670 tablet_copy_service.cc:161] P 9a3448a425c041258991dc60dd64ae7d: Beginning new tablet copy session on tablet 1983e0a53ee54b62afa16676bf986332 from peer 163649b5fbb140f18dccdf75f608b0f9 at {username='slave'} at 127.31.161.199:56747: session id = 163649b5fbb140f18dccdf75f608b0f9-1983e0a53ee54b62afa16676bf986332
I20240502 01:53:07.195740 2670 tablet_copy_source_session.cc:205] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:07.202610 4044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1983e0a53ee54b62afa16676bf986332. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:07.244633 4044 tablet_copy_client.cc:783] T 1983e0a53ee54b62afa16676bf986332 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:07.245137 4044 tablet_copy_client.cc:647] T 1983e0a53ee54b62afa16676bf986332 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:07.261303 4044 tablet_copy_client.cc:523] T 1983e0a53ee54b62afa16676bf986332 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:07.266000 4044 tablet_bootstrap.cc:492] T 1983e0a53ee54b62afa16676bf986332 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap starting.
I20240502 01:53:07.384779 4044 log.cc:824] T 1983e0a53ee54b62afa16676bf986332 P 163649b5fbb140f18dccdf75f608b0f9: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:07.386330 4077 ts_tablet_manager.cc:902] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9: Initiating tablet copy from peer 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585)
I20240502 01:53:07.390237 2384 consensus_queue.cc:237] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 1.85, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:07.403471 2649 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 1 index: 86. (index mismatch)
I20240502 01:53:07.404745 3865 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20240502 01:53:07.406410 2514 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 1 index: 86. (index mismatch)
I20240502 01:53:07.406999 3819 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20240502 01:53:07.415182 4060 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Committing config change with OpId 1.86: config changed from index -1 to 86, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:07.417685 2119 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 1 index: 86. (index mismatch)
I20240502 01:53:07.417598 2514 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.86: config changed from index -1 to 86, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:07.424618 1849 catalog_manager.cc:5484] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119 reported cstate change: config changed from index -1 to 86, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New cstate: current_term: 1 leader_uuid: "7ef43ea75ecb46ee905b781c91aed95a" committed_config { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:07.433724 3819 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20240502 01:53:07.439200 2124 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.86: config changed from index -1 to 86, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:07.447327 2649 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.86: config changed from index -1 to 86, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:07.473244 1989 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 1 index: 86. (index mismatch)
I20240502 01:53:07.474363 3819 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 85, Time since last communication: 0.000s
I20240502 01:53:07.503291 1989 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.86: config changed from index -1 to 86, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:07.536723 4088 ts_tablet_manager.cc:902] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906: Initiating tablet copy from peer 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987)
W20240502 01:53:07.575328 2321 consensus_peers.cc:485] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a -> Peer 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381): Couldn't send request to peer 163649b5fbb140f18dccdf75f608b0f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4220d1a63bd74ad0a5cac5a2f1bb988c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:07.615725 2253 consensus_queue.cc:237] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 1.86, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: true } }
I20240502 01:53:07.620609 2384 raft_consensus.cc:1261] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240502 01:53:07.624495 3707 consensus_queue.cc:1035] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20240502 01:53:07.625941 2649 raft_consensus.cc:1261] T 341e0200ea654dcfb91133135b458a16 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
W20240502 01:53:07.626474 2190 consensus_peers.cc:485] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 -> Peer 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621): Couldn't send request to peer 269c9061e1a1435e92f79f0b9486dfea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 341e0200ea654dcfb91133135b458a16. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:07.626928 4021 consensus_queue.cc:1035] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20240502 01:53:07.633164 1991 raft_consensus.cc:1261] T 341e0200ea654dcfb91133135b458a16 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240502 01:53:07.639001 3707 raft_consensus.cc:2932] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEADER]: Committing config change with OpId 1.87: config changed from index -1 to 87, NON_VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) added. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: true } } }
I20240502 01:53:07.642177 2384 raft_consensus.cc:2932] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index -1 to 87, NON_VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) added. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: true } } }
I20240502 01:53:07.646113 4021 consensus_queue.cc:1035] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 85, Time since last communication: 0.000s
I20240502 01:53:07.648664 4088 tablet_copy_client.cc:308] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.197:38987
I20240502 01:53:07.648947 1849 catalog_manager.cc:5484] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a reported cstate change: config changed from index -1 to 87, NON_VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) added. New cstate: current_term: 1 leader_uuid: "446c7f9bf31a4acc9dae8daf43330906" committed_config { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: true } } }
I20240502 01:53:07.651484 2649 raft_consensus.cc:2932] T 341e0200ea654dcfb91133135b458a16 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index -1 to 87, NON_VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) added. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: true } } }
I20240502 01:53:07.658476 1988 raft_consensus.cc:2932] T 341e0200ea654dcfb91133135b458a16 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index -1 to 87, NON_VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) added. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: true } } }
I20240502 01:53:07.672287 2517 raft_consensus.cc:1261] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240502 01:53:07.677863 2539 tablet_copy_service.cc:140] P 4d99e1fdda704211b3d101f892c57119: Received BeginTabletCopySession request for tablet e4672bc55a874f159bf10855f0698ba1 from peer 446c7f9bf31a4acc9dae8daf43330906 ({username='slave'} at 127.31.161.195:46969)
I20240502 01:53:07.678117 2539 tablet_copy_service.cc:161] P 4d99e1fdda704211b3d101f892c57119: Beginning new tablet copy session on tablet e4672bc55a874f159bf10855f0698ba1 from peer 446c7f9bf31a4acc9dae8daf43330906 at {username='slave'} at 127.31.161.195:46969: session id = 446c7f9bf31a4acc9dae8daf43330906-e4672bc55a874f159bf10855f0698ba1
I20240502 01:53:07.684209 2539 tablet_copy_source_session.cc:205] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:07.690732 4044 tablet_bootstrap.cc:492] T 1983e0a53ee54b62afa16676bf986332 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=78 overwritten=0 applied=78 ignored=0} inserts{seen=709 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:07.691473 4044 tablet_bootstrap.cc:492] T 1983e0a53ee54b62afa16676bf986332 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap complete.
I20240502 01:53:07.692075 4044 ts_tablet_manager.cc:1367] T 1983e0a53ee54b62afa16676bf986332 P 163649b5fbb140f18dccdf75f608b0f9: Time spent bootstrapping tablet: real 0.426s user 0.141s sys 0.010s
I20240502 01:53:07.697175 3707 consensus_queue.cc:1035] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20240502 01:53:07.701565 4044 raft_consensus.cc:348] T 1983e0a53ee54b62afa16676bf986332 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:07.702226 4044 raft_consensus.cc:729] T 1983e0a53ee54b62afa16676bf986332 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 163649b5fbb140f18dccdf75f608b0f9, State: Initialized, Role: LEARNER
W20240502 01:53:07.703016 2189 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 43861.7us
I20240502 01:53:07.702932 4044 consensus_queue.cc:260] T 1983e0a53ee54b62afa16676bf986332 P 163649b5fbb140f18dccdf75f608b0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 78, Last appended: 1.78, Last appended by leader: 78, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:07.706651 4088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4672bc55a874f159bf10855f0698ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:07.715126 4044 ts_tablet_manager.cc:1398] T 1983e0a53ee54b62afa16676bf986332 P 163649b5fbb140f18dccdf75f608b0f9: Time spent starting tablet: real 0.022s user 0.009s sys 0.006s
I20240502 01:53:07.722016 2670 tablet_copy_service.cc:342] P 9a3448a425c041258991dc60dd64ae7d: Request end of tablet copy session 163649b5fbb140f18dccdf75f608b0f9-1983e0a53ee54b62afa16676bf986332 received from {username='slave'} at 127.31.161.199:56747
I20240502 01:53:07.722206 2670 tablet_copy_service.cc:433] P 9a3448a425c041258991dc60dd64ae7d: ending tablet copy session 163649b5fbb140f18dccdf75f608b0f9-1983e0a53ee54b62afa16676bf986332 on tablet 1983e0a53ee54b62afa16676bf986332 with peer 163649b5fbb140f18dccdf75f608b0f9
I20240502 01:53:07.738507 2514 raft_consensus.cc:2932] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index -1 to 87, NON_VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) added. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: true } } }
I20240502 01:53:07.743846 4077 tablet_copy_client.cc:308] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.198:37585
I20240502 01:53:07.754307 2670 tablet_copy_service.cc:140] P 9a3448a425c041258991dc60dd64ae7d: Received BeginTabletCopySession request for tablet e567c877188549fc965637e89bf437ae from peer 163649b5fbb140f18dccdf75f608b0f9 ({username='slave'} at 127.31.161.199:56747)
I20240502 01:53:07.754511 2670 tablet_copy_service.cc:161] P 9a3448a425c041258991dc60dd64ae7d: Beginning new tablet copy session on tablet e567c877188549fc965637e89bf437ae from peer 163649b5fbb140f18dccdf75f608b0f9 at {username='slave'} at 127.31.161.199:56747: session id = 163649b5fbb140f18dccdf75f608b0f9-e567c877188549fc965637e89bf437ae
I20240502 01:53:07.759841 2670 tablet_copy_source_session.cc:205] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:07.767186 4077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e567c877188549fc965637e89bf437ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:07.809943 4077 tablet_copy_client.cc:783] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:07.810351 4077 tablet_copy_client.cc:647] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:07.857821 4077 tablet_copy_client.cc:523] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:07.943754 3605 raft_consensus.cc:1203] T 1983e0a53ee54b62afa16676bf986332 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.73->[1.74-1.85] Dedup: 1.78->[1.79-1.85]
I20240502 01:53:07.946349 4088 tablet_copy_client.cc:783] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:07.946707 4088 tablet_copy_client.cc:647] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:07.953580 4077 tablet_bootstrap.cc:492] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap starting.
I20240502 01:53:07.958083 4088 tablet_copy_client.cc:523] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:07.961825 2251 consensus_queue.cc:237] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 1.64, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:07.978389 2383 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240502 01:53:07.979528 2518 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240502 01:53:07.979616 3980 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20240502 01:53:07.981931 3707 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20240502 01:53:07.984489 2648 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240502 01:53:07.985349 3707 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20240502 01:53:07.989228 3980 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEADER]: Committing config change with OpId 1.65: config changed from index -1 to 65, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:07.993640 2385 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index -1 to 65, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:07.993896 2648 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index -1 to 65, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:08.001612 1844 catalog_manager.cc:5484] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 reported cstate change: config changed from index -1 to 65, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New cstate: current_term: 1 leader_uuid: "446c7f9bf31a4acc9dae8daf43330906" committed_config { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:08.005394 2516 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index -1 to 65, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:08.029203 4044 ts_tablet_manager.cc:902] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9: Initiating tablet copy from peer 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225)
I20240502 01:53:08.081041 1988 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240502 01:53:08.093127 4088 tablet_bootstrap.cc:492] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906: Bootstrap starting.
I20240502 01:53:08.091763 4015 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 63, Time since last communication: 0.001s
I20240502 01:53:08.155542 1989 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index -1 to 65, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:08.204479 4121 ts_tablet_manager.cc:902] T 341e0200ea654dcfb91133135b458a16 P 269c9061e1a1435e92f79f0b9486dfea: Initiating tablet copy from peer 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353)
W20240502 01:53:08.205853 2189 consensus_peers.cc:485] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 -> Peer 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381): Couldn't send request to peer 163649b5fbb140f18dccdf75f608b0f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4a07f0fe1964f958ad7ca4a63d054a6. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:08.279752 4044 tablet_copy_client.cc:308] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:39225
I20240502 01:53:08.280517 3932 raft_consensus.cc:1050] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d: attempting to promote NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 to VOTER
I20240502 01:53:08.282549 3932 consensus_queue.cc:237] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 1.86, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } }
I20240502 01:53:08.301437 2124 raft_consensus.cc:1261] T 1983e0a53ee54b62afa16676bf986332 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240502 01:53:08.301985 3798 consensus_queue.cc:1035] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20240502 01:53:08.304342 4121 tablet_copy_client.cc:308] T 341e0200ea654dcfb91133135b458a16 P 269c9061e1a1435e92f79f0b9486dfea: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:39353
I20240502 01:53:08.305279 2253 raft_consensus.cc:1261] T 1983e0a53ee54b62afa16676bf986332 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240502 01:53:08.305804 3932 consensus_queue.cc:1035] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20240502 01:53:08.306617 2518 raft_consensus.cc:1261] T 1983e0a53ee54b62afa16676bf986332 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240502 01:53:08.308393 3932 consensus_queue.cc:1035] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20240502 01:53:08.311686 2275 tablet_copy_service.cc:140] P 446c7f9bf31a4acc9dae8daf43330906: Received BeginTabletCopySession request for tablet 341e0200ea654dcfb91133135b458a16 from peer 269c9061e1a1435e92f79f0b9486dfea ({username='slave'} at 127.31.161.194:49395)
I20240502 01:53:08.311975 2275 tablet_copy_service.cc:161] P 446c7f9bf31a4acc9dae8daf43330906: Beginning new tablet copy session on tablet 341e0200ea654dcfb91133135b458a16 from peer 269c9061e1a1435e92f79f0b9486dfea at {username='slave'} at 127.31.161.194:49395: session id = 269c9061e1a1435e92f79f0b9486dfea-341e0200ea654dcfb91133135b458a16
I20240502 01:53:08.319185 2275 tablet_copy_source_session.cc:205] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:08.328974 4121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 341e0200ea654dcfb91133135b458a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:08.341626 2383 raft_consensus.cc:1261] T 1983e0a53ee54b62afa16676bf986332 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240502 01:53:08.346086 3798 raft_consensus.cc:2932] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [term 1 LEADER]: Committing config change with OpId 1.87: config changed from index 74 to 87, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:08.346845 2119 raft_consensus.cc:2932] T 1983e0a53ee54b62afa16676bf986332 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index 74 to 87, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:08.353977 1849 catalog_manager.cc:5484] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d reported cstate change: config changed from index 74 to 87, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9a3448a425c041258991dc60dd64ae7d" committed_config { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:08.356287 3933 consensus_queue.cc:1035] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20240502 01:53:08.361555 2253 raft_consensus.cc:2932] T 1983e0a53ee54b62afa16676bf986332 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index 74 to 87, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:08.369038 2518 raft_consensus.cc:2932] T 1983e0a53ee54b62afa16676bf986332 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index 74 to 87, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:08.368244 2383 raft_consensus.cc:2932] T 1983e0a53ee54b62afa16676bf986332 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index 74 to 87, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:08.413852 4137 ts_tablet_manager.cc:902] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9: Initiating tablet copy from peer 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353)
I20240502 01:53:08.414013 3604 raft_consensus.cc:1261] T 1983e0a53ee54b62afa16676bf986332 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240502 01:53:08.432781 3798 consensus_queue.cc:1035] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20240502 01:53:08.438441 3604 raft_consensus.cc:2932] T 1983e0a53ee54b62afa16676bf986332 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index 74 to 87, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:08.467876 4121 tablet_copy_client.cc:783] T 341e0200ea654dcfb91133135b458a16 P 269c9061e1a1435e92f79f0b9486dfea: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:08.473218 4121 tablet_copy_client.cc:647] T 341e0200ea654dcfb91133135b458a16 P 269c9061e1a1435e92f79f0b9486dfea: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:08.560122 4121 tablet_copy_client.cc:523] T 341e0200ea654dcfb91133135b458a16 P 269c9061e1a1435e92f79f0b9486dfea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:08.560724 2645 consensus_queue.cc:237] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 1.90, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } }
I20240502 01:53:08.582319 3605 raft_consensus.cc:1261] T 1983e0a53ee54b62afa16676bf986332 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 87. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20240502 01:53:08.583122 4071 consensus_queue.cc:1035] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 87, Time since last communication: 0.000s
I20240502 01:53:08.583748 2516 raft_consensus.cc:1261] T 1983e0a53ee54b62afa16676bf986332 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20240502 01:53:08.585384 4071 consensus_queue.cc:1035] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 89, Time since last communication: 0.000s
I20240502 01:53:08.589493 4121 tablet_bootstrap.cc:492] T 341e0200ea654dcfb91133135b458a16 P 269c9061e1a1435e92f79f0b9486dfea: Bootstrap starting.
I20240502 01:53:08.603739 4038 raft_consensus.cc:2932] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [term 1 LEADER]: Committing config change with OpId 1.91: config changed from index 87 to 91, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:08.607451 1834 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1983e0a53ee54b62afa16676bf986332 with cas_config_opid_index 87: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:08.610613 2515 raft_consensus.cc:2932] T 1983e0a53ee54b62afa16676bf986332 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 87 to 91, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:08.611018 1849 catalog_manager.cc:5484] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d reported cstate change: config changed from index 87 to 91, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "9a3448a425c041258991dc60dd64ae7d" committed_config { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:08.626538 3605 raft_consensus.cc:2932] T 1983e0a53ee54b62afa16676bf986332 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 87 to 91, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:08.631302 3604 raft_consensus.cc:1203] T 1983e0a53ee54b62afa16676bf986332 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.87->[1.88-1.90] Dedup: 1.90->[]
I20240502 01:53:08.642134 2249 raft_consensus.cc:1261] T 1983e0a53ee54b62afa16676bf986332 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20240502 01:53:08.643532 4038 consensus_queue.cc:1035] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 89, Time since last communication: 0.000s
I20240502 01:53:08.645152 2249 raft_consensus.cc:2932] T 1983e0a53ee54b62afa16676bf986332 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 87 to 91, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:08.647022 2119 raft_consensus.cc:1261] T 1983e0a53ee54b62afa16676bf986332 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20240502 01:53:08.647631 4038 consensus_queue.cc:1035] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 89, Time since last communication: 0.000s
I20240502 01:53:08.755883 2124 raft_consensus.cc:2932] T 1983e0a53ee54b62afa16676bf986332 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 87 to 91, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:08.803412 2408 tablet_copy_service.cc:140] P 7ef43ea75ecb46ee905b781c91aed95a: Received BeginTabletCopySession request for tablet 4220d1a63bd74ad0a5cac5a2f1bb988c from peer 163649b5fbb140f18dccdf75f608b0f9 ({username='slave'} at 127.31.161.199:46507)
I20240502 01:53:08.803694 2408 tablet_copy_service.cc:161] P 7ef43ea75ecb46ee905b781c91aed95a: Beginning new tablet copy session on tablet 4220d1a63bd74ad0a5cac5a2f1bb988c from peer 163649b5fbb140f18dccdf75f608b0f9 at {username='slave'} at 127.31.161.199:46507: session id = 163649b5fbb140f18dccdf75f608b0f9-4220d1a63bd74ad0a5cac5a2f1bb988c
I20240502 01:53:08.807327 2408 tablet_copy_source_session.cc:205] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:08.835520 4044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4220d1a63bd74ad0a5cac5a2f1bb988c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:08.843619 4044 tablet_copy_client.cc:783] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:08.843901 4044 tablet_copy_client.cc:647] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:08.859568 4044 tablet_copy_client.cc:523] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:08.861394 2366 tablet_service.cc:1514] Processing DeleteTablet for tablet 1983e0a53ee54b62afa16676bf986332 with delete_type TABLET_DATA_TOMBSTONED (TS 7ef43ea75ecb46ee905b781c91aed95a not found in new config with opid_index 91) from {username='slave'} at 127.0.0.1:54148
I20240502 01:53:08.888211 4157 tablet_replica.cc:330] T 1983e0a53ee54b62afa16676bf986332 P 7ef43ea75ecb46ee905b781c91aed95a: stopping tablet replica
I20240502 01:53:08.886600 4044 tablet_bootstrap.cc:492] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap starting.
I20240502 01:53:08.899099 4157 raft_consensus.cc:2229] T 1983e0a53ee54b62afa16676bf986332 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:08.903687 4157 raft_consensus.cc:2258] T 1983e0a53ee54b62afa16676bf986332 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:08.907338 4157 ts_tablet_manager.cc:1875] T 1983e0a53ee54b62afa16676bf986332 P 7ef43ea75ecb46ee905b781c91aed95a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:08.930512 4157 ts_tablet_manager.cc:1888] T 1983e0a53ee54b62afa16676bf986332 P 7ef43ea75ecb46ee905b781c91aed95a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.89
I20240502 01:53:08.930773 4157 log.cc:1196] T 1983e0a53ee54b62afa16676bf986332 P 7ef43ea75ecb46ee905b781c91aed95a: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/1983e0a53ee54b62afa16676bf986332
I20240502 01:53:08.933779 1835 catalog_manager.cc:4830] TS 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225): tablet 1983e0a53ee54b62afa16676bf986332 (table rebalance_test_table_3 [id=3402138c5f114264ac2eb0f198dbe6b3]) successfully deleted
W20240502 01:53:08.954439 3526 kernel_stack_watchdog.cc:198] Thread 4137 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 366ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f49339d6980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7f4930644d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7f49306657ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f493065debb kudu::ThreadPool::Init() at ??:0
@ 0x7f493065dcf9 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f493fab08de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f493fac8daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f493fbf3480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f493fc2ba8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f493fc0e981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f493fc0e70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f4930667b75 kudu::ThreadPool::DispatchThread() at ??:0
I20240502 01:53:08.988209 4121 tablet_bootstrap.cc:492] T 341e0200ea654dcfb91133135b458a16 P 269c9061e1a1435e92f79f0b9486dfea: Bootstrap replayed 1/1 log segments. Stats: ops{read=91 overwritten=0 applied=91 ignored=0} inserts{seen=776 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:08.988757 4121 tablet_bootstrap.cc:492] T 341e0200ea654dcfb91133135b458a16 P 269c9061e1a1435e92f79f0b9486dfea: Bootstrap complete.
I20240502 01:53:08.989200 4121 ts_tablet_manager.cc:1367] T 341e0200ea654dcfb91133135b458a16 P 269c9061e1a1435e92f79f0b9486dfea: Time spent bootstrapping tablet: real 0.400s user 0.130s sys 0.025s
I20240502 01:53:08.990031 4121 raft_consensus.cc:348] T 341e0200ea654dcfb91133135b458a16 P 269c9061e1a1435e92f79f0b9486dfea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: true } }
I20240502 01:53:08.990324 4121 raft_consensus.cc:729] T 341e0200ea654dcfb91133135b458a16 P 269c9061e1a1435e92f79f0b9486dfea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 269c9061e1a1435e92f79f0b9486dfea, State: Initialized, Role: LEARNER
I20240502 01:53:08.990550 4121 consensus_queue.cc:260] T 341e0200ea654dcfb91133135b458a16 P 269c9061e1a1435e92f79f0b9486dfea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 91, Last appended: 1.91, Last appended by leader: 91, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: true } }
I20240502 01:53:08.991252 4121 ts_tablet_manager.cc:1398] T 341e0200ea654dcfb91133135b458a16 P 269c9061e1a1435e92f79f0b9486dfea: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:08.992352 2275 tablet_copy_service.cc:342] P 446c7f9bf31a4acc9dae8daf43330906: Request end of tablet copy session 269c9061e1a1435e92f79f0b9486dfea-341e0200ea654dcfb91133135b458a16 received from {username='slave'} at 127.31.161.194:49395
I20240502 01:53:08.992558 2275 tablet_copy_service.cc:433] P 446c7f9bf31a4acc9dae8daf43330906: ending tablet copy session 269c9061e1a1435e92f79f0b9486dfea-341e0200ea654dcfb91133135b458a16 on tablet 341e0200ea654dcfb91133135b458a16 with peer 269c9061e1a1435e92f79f0b9486dfea
I20240502 01:53:09.002372 2385 consensus_queue.cc:237] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 1.81, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
W20240502 01:53:09.006654 2321 consensus_peers.cc:485] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a -> Peer 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381): Couldn't send request to peer 163649b5fbb140f18dccdf75f608b0f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c291b2a5e9d140f881b2bd54f8c9ed19. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:09.007308 1987 raft_consensus.cc:1261] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240502 01:53:09.007938 3718 consensus_queue.cc:1035] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 80, Time since last communication: 0.000s
I20240502 01:53:09.018975 2249 raft_consensus.cc:1261] T c291b2a5e9d140f881b2bd54f8c9ed19 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240502 01:53:09.019629 3718 consensus_queue.cc:1035] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 80, Time since last communication: 0.000s
I20240502 01:53:09.070426 2518 raft_consensus.cc:1261] T c291b2a5e9d140f881b2bd54f8c9ed19 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240502 01:53:09.119261 3865 raft_consensus.cc:2932] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Committing config change with OpId 1.82: config changed from index -1 to 82, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:09.159165 1987 raft_consensus.cc:2932] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index -1 to 82, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:09.164912 1849 catalog_manager.cc:5484] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a reported cstate change: config changed from index -1 to 82, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New cstate: current_term: 1 leader_uuid: "7ef43ea75ecb46ee905b781c91aed95a" committed_config { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:53:09.201531 2322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 67493.6us
I20240502 01:53:09.251264 2123 raft_consensus.cc:1261] T c291b2a5e9d140f881b2bd54f8c9ed19 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240502 01:53:09.270504 3865 consensus_queue.cc:1035] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 79, Time since last communication: 0.000s
I20240502 01:53:09.271486 2251 raft_consensus.cc:2932] T c291b2a5e9d140f881b2bd54f8c9ed19 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index -1 to 82, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
W20240502 01:53:09.327464 4137 thread.cc:636] tablet-download-pool-f4a07f0fe1964f958ad7ca4a63d054a6 [worker] (thread pool) Time spent creating pthread: real 0.913s user 0.076s sys 0.076s
W20240502 01:53:09.327628 4137 thread.cc:602] tablet-download-pool-f4a07f0fe1964f958ad7ca4a63d054a6 [worker] (thread pool) Time spent starting thread: real 0.913s user 0.076s sys 0.076s
I20240502 01:53:09.366698 2516 raft_consensus.cc:2932] T c291b2a5e9d140f881b2bd54f8c9ed19 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index -1 to 82, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:09.400970 3879 consensus_queue.cc:1035] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 80, Time since last communication: 0.000s
W20240502 01:53:09.405349 2322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 57007.5us
W20240502 01:53:09.561946 2454 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 53837.4us
I20240502 01:53:09.570308 2121 raft_consensus.cc:2932] T c291b2a5e9d140f881b2bd54f8c9ed19 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index -1 to 82, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:09.586902 2121 raft_consensus.cc:1203] T 341e0200ea654dcfb91133135b458a16 P 269c9061e1a1435e92f79f0b9486dfea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.86->[1.87-1.106] Dedup: 1.91->[1.92-1.106]
I20240502 01:53:09.682942 4207 ts_tablet_manager.cc:902] T c291b2a5e9d140f881b2bd54f8c9ed19 P 163649b5fbb140f18dccdf75f608b0f9: Initiating tablet copy from peer 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225)
I20240502 01:53:09.701803 4207 tablet_copy_client.cc:308] T c291b2a5e9d140f881b2bd54f8c9ed19 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:39225
I20240502 01:53:09.702915 2408 tablet_copy_service.cc:140] P 7ef43ea75ecb46ee905b781c91aed95a: Received BeginTabletCopySession request for tablet c291b2a5e9d140f881b2bd54f8c9ed19 from peer 163649b5fbb140f18dccdf75f608b0f9 ({username='slave'} at 127.31.161.199:46507)
I20240502 01:53:09.703086 2408 tablet_copy_service.cc:161] P 7ef43ea75ecb46ee905b781c91aed95a: Beginning new tablet copy session on tablet c291b2a5e9d140f881b2bd54f8c9ed19 from peer 163649b5fbb140f18dccdf75f608b0f9 at {username='slave'} at 127.31.161.199:46507: session id = 163649b5fbb140f18dccdf75f608b0f9-c291b2a5e9d140f881b2bd54f8c9ed19
I20240502 01:53:09.706171 2408 tablet_copy_source_session.cc:205] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:09.707470 4207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c291b2a5e9d140f881b2bd54f8c9ed19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:09.713819 4207 tablet_copy_client.cc:783] T c291b2a5e9d140f881b2bd54f8c9ed19 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:09.714025 4207 tablet_copy_client.cc:647] T c291b2a5e9d140f881b2bd54f8c9ed19 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:09.715857 4207 tablet_copy_client.cc:523] T c291b2a5e9d140f881b2bd54f8c9ed19 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:09.719087 4207 tablet_bootstrap.cc:492] T c291b2a5e9d140f881b2bd54f8c9ed19 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap starting.
W20240502 01:53:09.743666 2322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 66072.7us
W20240502 01:53:09.744172 4077 log.cc:924] Time spent T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
W20240502 01:53:09.750085 2585 consensus_peers.cc:485] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d -> Peer 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381): Couldn't send request to peer 163649b5fbb140f18dccdf75f608b0f9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240502 01:53:09.825280 4137 tablet_copy_client.cc:308] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:39353
I20240502 01:53:09.852180 4020 raft_consensus.cc:1050] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906: attempting to promote NON_VOTER 269c9061e1a1435e92f79f0b9486dfea to VOTER
I20240502 01:53:09.853401 4020 consensus_queue.cc:237] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.107, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: false } }
I20240502 01:53:09.856770 2119 raft_consensus.cc:1261] T 341e0200ea654dcfb91133135b458a16 P 269c9061e1a1435e92f79f0b9486dfea [term 1 LEARNER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240502 01:53:09.857522 4015 consensus_queue.cc:1035] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 106, Time since last communication: 0.000s
I20240502 01:53:09.868988 2385 raft_consensus.cc:1261] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240502 01:53:09.869572 4020 consensus_queue.cc:1035] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 106, Time since last communication: 0.000s
I20240502 01:53:09.883319 2644 raft_consensus.cc:1261] T 341e0200ea654dcfb91133135b458a16 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240502 01:53:09.894812 1987 raft_consensus.cc:1261] T 341e0200ea654dcfb91133135b458a16 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240502 01:53:09.922515 4015 consensus_queue.cc:1035] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 106, Time since last communication: 0.001s
I20240502 01:53:09.923022 4015 consensus_queue.cc:1035] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 106, Time since last communication: 0.000s
I20240502 01:53:09.929124 4088 tablet_bootstrap.cc:492] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906: Bootstrap replayed 1/1 log segments. Stats: ops{read=85 overwritten=0 applied=85 ignored=0} inserts{seen=757 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:09.930908 4088 tablet_bootstrap.cc:492] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906: Bootstrap complete.
I20240502 01:53:09.930615 3980 raft_consensus.cc:2932] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEADER]: Committing config change with OpId 1.108: config changed from index 87 to 108, 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) changed from NON_VOTER to VOTER. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: false } } }
I20240502 01:53:09.931447 4088 ts_tablet_manager.cc:1367] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906: Time spent bootstrapping tablet: real 1.839s user 0.163s sys 0.074s
I20240502 01:53:09.933935 4088 raft_consensus.cc:348] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: true } }
I20240502 01:53:09.934196 4088 raft_consensus.cc:729] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 446c7f9bf31a4acc9dae8daf43330906, State: Initialized, Role: LEARNER
I20240502 01:53:09.934422 4088 consensus_queue.cc:260] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 85, Last appended: 1.85, Last appended by leader: 85, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: true } }
I20240502 01:53:09.935165 4088 ts_tablet_manager.cc:1398] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:53:09.940732 2275 tablet_copy_service.cc:140] P 446c7f9bf31a4acc9dae8daf43330906: Received BeginTabletCopySession request for tablet f4a07f0fe1964f958ad7ca4a63d054a6 from peer 163649b5fbb140f18dccdf75f608b0f9 ({username='slave'} at 127.31.161.199:59365)
I20240502 01:53:09.940904 2275 tablet_copy_service.cc:161] P 446c7f9bf31a4acc9dae8daf43330906: Beginning new tablet copy session on tablet f4a07f0fe1964f958ad7ca4a63d054a6 from peer 163649b5fbb140f18dccdf75f608b0f9 at {username='slave'} at 127.31.161.199:59365: session id = 163649b5fbb140f18dccdf75f608b0f9-f4a07f0fe1964f958ad7ca4a63d054a6
I20240502 01:53:09.945894 1987 raft_consensus.cc:2932] T 341e0200ea654dcfb91133135b458a16 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index 87 to 108, 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) changed from NON_VOTER to VOTER. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: false } } }
I20240502 01:53:09.959210 2275 tablet_copy_source_session.cc:205] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:09.960634 1848 catalog_manager.cc:5484] T 341e0200ea654dcfb91133135b458a16 P 58dae5dc94444d5e92e6907c5b637879 reported cstate change: config changed from index 87 to 108, 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "446c7f9bf31a4acc9dae8daf43330906" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: false } } }
I20240502 01:53:09.964543 4137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a07f0fe1964f958ad7ca4a63d054a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:09.964463 2644 raft_consensus.cc:2932] T 341e0200ea654dcfb91133135b458a16 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index 87 to 108, 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) changed from NON_VOTER to VOTER. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: false } } }
I20240502 01:53:09.980120 4137 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.31.161.195:49113 (request call id 1) took 1585 ms. Trace:
I20240502 01:53:09.980307 4137 rpcz_store.cc:276] 0502 01:53:08.394859 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 01:53:08.394924 (+ 65us) service_pool.cc:225] Handling call
0502 01:53:08.414072 (+ 19148us) ts_tablet_manager.cc:903] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9: Initiating tablet copy from peer 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353)
0502 01:53:09.964026 (+1549954us) tablet_copy_client.cc:353] Tablet copy session begun
0502 01:53:09.967742 (+ 3716us) tablet_metadata.cc:641] Metadata flushed
0502 01:53:09.967864 (+ 122us) tablet_copy_client.cc:482] Wrote new tablet metadata
0502 01:53:09.979813 (+ 11949us) tablet_copy_client.cc:403] Wrote new consensus metadata
0502 01:53:09.979934 (+ 121us) tablet_replica.cc:205] Creating consensus instance
0502 01:53:09.980033 (+ 99us) ts_tablet_manager.cc:958] Replica registered.
0502 01:53:09.980060 (+ 27us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":88,"spinlock_wait_cycles":3328,"tablet-copy.queue_time_us":18597,"thread_start_us":1428934,"threads_started":5}
I20240502 01:53:09.983776 4137 tablet_copy_client.cc:783] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:09.984360 4137 tablet_copy_client.cc:647] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:09.996732 4077 tablet_bootstrap.cc:492] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=56 overwritten=0 applied=56 ignored=0} inserts{seen=687 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:10.010921 2119 raft_consensus.cc:2932] T 341e0200ea654dcfb91133135b458a16 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index 87 to 108, 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) changed from NON_VOTER to VOTER. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: false } } }
I20240502 01:53:10.025475 4077 tablet_bootstrap.cc:492] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap complete.
I20240502 01:53:10.029172 4077 ts_tablet_manager.cc:1367] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9: Time spent bootstrapping tablet: real 2.076s user 0.139s sys 0.058s
I20240502 01:53:10.030287 4077 raft_consensus.cc:348] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:10.030551 4077 raft_consensus.cc:729] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 163649b5fbb140f18dccdf75f608b0f9, State: Initialized, Role: LEARNER
I20240502 01:53:10.030778 4077 consensus_queue.cc:260] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 56, Last appended: 1.56, Last appended by leader: 56, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:10.031805 4137 tablet_copy_client.cc:523] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:10.037238 4137 tablet_bootstrap.cc:492] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap starting.
W20240502 01:53:10.041620 2454 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 51064.5us
I20240502 01:53:10.067893 2515 raft_consensus.cc:1261] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240502 01:53:10.077817 2539 tablet_copy_service.cc:342] P 4d99e1fdda704211b3d101f892c57119: Request end of tablet copy session 446c7f9bf31a4acc9dae8daf43330906-e4672bc55a874f159bf10855f0698ba1 received from {username='slave'} at 127.31.161.195:46969
I20240502 01:53:10.077998 2539 tablet_copy_service.cc:433] P 4d99e1fdda704211b3d101f892c57119: ending tablet copy session 446c7f9bf31a4acc9dae8daf43330906-e4672bc55a874f159bf10855f0698ba1 on tablet e4672bc55a874f159bf10855f0698ba1 with peer 446c7f9bf31a4acc9dae8daf43330906
I20240502 01:53:10.093046 4077 ts_tablet_manager.cc:1398] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9: Time spent starting tablet: real 0.064s user 0.002s sys 0.000s
I20240502 01:53:10.094849 2670 tablet_copy_service.cc:342] P 9a3448a425c041258991dc60dd64ae7d: Request end of tablet copy session 163649b5fbb140f18dccdf75f608b0f9-e567c877188549fc965637e89bf437ae received from {username='slave'} at 127.31.161.199:56747
I20240502 01:53:10.094969 2670 tablet_copy_service.cc:433] P 9a3448a425c041258991dc60dd64ae7d: ending tablet copy session 163649b5fbb140f18dccdf75f608b0f9-e567c877188549fc965637e89bf437ae on tablet e567c877188549fc965637e89bf437ae with peer 163649b5fbb140f18dccdf75f608b0f9
W20240502 01:53:10.105942 2188 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.196:39225, user_credentials={real_user=slave}} blocked reactor thread for 130004us
I20240502 01:53:10.106798 2385 raft_consensus.cc:2932] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index 87 to 108, 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) changed from NON_VOTER to VOTER. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: false } } }
I20240502 01:53:10.135406 4170 consensus_queue.cc:1035] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 106, Time since last communication: 0.000s
I20240502 01:53:10.160626 2251 raft_consensus.cc:1203] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.78->[1.79-1.100] Dedup: 1.85->[1.86-1.100]
I20240502 01:53:10.157049 4044 tablet_bootstrap.cc:492] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=101 overwritten=0 applied=101 ignored=0} inserts{seen=786 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:10.167328 4044 tablet_bootstrap.cc:492] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap complete.
I20240502 01:53:10.167639 4044 ts_tablet_manager.cc:1367] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9: Time spent bootstrapping tablet: real 1.281s user 0.148s sys 0.017s
I20240502 01:53:10.168450 4044 raft_consensus.cc:348] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:10.168752 4044 raft_consensus.cc:729] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 163649b5fbb140f18dccdf75f608b0f9, State: Initialized, Role: LEARNER
I20240502 01:53:10.168984 4044 consensus_queue.cc:260] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 101, Last appended: 1.101, Last appended by leader: 101, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:10.193431 4044 ts_tablet_manager.cc:1398] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9: Time spent starting tablet: real 0.026s user 0.003s sys 0.000s
I20240502 01:53:10.208375 2408 tablet_copy_service.cc:342] P 7ef43ea75ecb46ee905b781c91aed95a: Request end of tablet copy session 163649b5fbb140f18dccdf75f608b0f9-4220d1a63bd74ad0a5cac5a2f1bb988c received from {username='slave'} at 127.31.161.199:46507
I20240502 01:53:10.208536 2408 tablet_copy_service.cc:433] P 7ef43ea75ecb46ee905b781c91aed95a: ending tablet copy session 163649b5fbb140f18dccdf75f608b0f9-4220d1a63bd74ad0a5cac5a2f1bb988c on tablet 4220d1a63bd74ad0a5cac5a2f1bb988c with peer 163649b5fbb140f18dccdf75f608b0f9
I20240502 01:53:10.229386 2517 raft_consensus.cc:2932] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index 87 to 108, 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) changed from NON_VOTER to VOTER. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: false } } }
I20240502 01:53:10.298839 3604 raft_consensus.cc:1203] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.85->[1.86-1.109] Dedup: 1.101->[1.102-1.109]
I20240502 01:53:10.322846 3605 raft_consensus.cc:1203] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.50->[1.51-1.75] Dedup: 1.56->[1.57-1.75]
I20240502 01:53:10.377163 3798 raft_consensus.cc:1050] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d: attempting to promote NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 to VOTER
I20240502 01:53:10.378155 3798 consensus_queue.cc:237] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 1.76, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } }
I20240502 01:53:10.380745 2124 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 77. (index mismatch)
I20240502 01:53:10.381330 4071 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240502 01:53:10.387252 3603 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 78. (index mismatch)
I20240502 01:53:10.387820 4038 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240502 01:53:10.392238 1991 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 78. (index mismatch)
I20240502 01:53:10.392668 3798 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240502 01:53:10.399042 4071 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [term 1 LEADER]: Committing config change with OpId 1.77: config changed from index 51 to 77, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:10.403439 3603 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index 51 to 77, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:10.404487 2387 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 78. (index mismatch)
I20240502 01:53:10.430094 2124 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index 51 to 77, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:10.438242 1848 catalog_manager.cc:5484] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea reported cstate change: config changed from index 51 to 77, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9a3448a425c041258991dc60dd64ae7d" committed_config { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:10.452613 1991 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index 51 to 77, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:10.481031 4249 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 69, Time since last communication: 0.000s
I20240502 01:53:10.488265 4207 tablet_bootstrap.cc:492] T c291b2a5e9d140f881b2bd54f8c9ed19 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=83 overwritten=0 applied=82 ignored=0} inserts{seen=767 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:53:10.489650 2249 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 74. Preceding OpId from leader: term: 1 index: 78. (index mismatch)
I20240502 01:53:10.505681 4207 tablet_bootstrap.cc:492] T c291b2a5e9d140f881b2bd54f8c9ed19 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap complete.
I20240502 01:53:10.506845 4207 ts_tablet_manager.cc:1367] T c291b2a5e9d140f881b2bd54f8c9ed19 P 163649b5fbb140f18dccdf75f608b0f9: Time spent bootstrapping tablet: real 0.788s user 0.123s sys 0.020s
I20240502 01:53:10.508257 4249 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 73, Time since last communication: 0.000s
I20240502 01:53:10.510612 4207 raft_consensus.cc:348] T c291b2a5e9d140f881b2bd54f8c9ed19 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:10.513854 2249 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index 51 to 77, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:10.514878 4207 raft_consensus.cc:729] T c291b2a5e9d140f881b2bd54f8c9ed19 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 163649b5fbb140f18dccdf75f608b0f9, State: Initialized, Role: LEARNER
I20240502 01:53:10.515391 4207 consensus_queue.cc:260] T c291b2a5e9d140f881b2bd54f8c9ed19 P 163649b5fbb140f18dccdf75f608b0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 82, Last appended: 1.83, Last appended by leader: 83, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:10.516711 4207 ts_tablet_manager.cc:1398] T c291b2a5e9d140f881b2bd54f8c9ed19 P 163649b5fbb140f18dccdf75f608b0f9: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:53:10.517966 2408 tablet_copy_service.cc:342] P 7ef43ea75ecb46ee905b781c91aed95a: Request end of tablet copy session 163649b5fbb140f18dccdf75f608b0f9-c291b2a5e9d140f881b2bd54f8c9ed19 received from {username='slave'} at 127.31.161.199:46507
I20240502 01:53:10.518124 2408 tablet_copy_service.cc:433] P 7ef43ea75ecb46ee905b781c91aed95a: ending tablet copy session 163649b5fbb140f18dccdf75f608b0f9-c291b2a5e9d140f881b2bd54f8c9ed19 on tablet c291b2a5e9d140f881b2bd54f8c9ed19 with peer 163649b5fbb140f18dccdf75f608b0f9
I20240502 01:53:10.714782 3865 raft_consensus.cc:1050] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a: attempting to promote NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 to VOTER
I20240502 01:53:10.716336 3865 consensus_queue.cc:237] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 1.111, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } }
I20240502 01:53:10.720253 1991 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 1 index: 112. (index mismatch)
I20240502 01:53:10.724869 2119 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 1 index: 112. (index mismatch)
I20240502 01:53:10.730015 3865 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 110, Time since last communication: 0.000s
I20240502 01:53:10.737160 2648 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 111. Preceding OpId from leader: term: 1 index: 112. (index mismatch)
I20240502 01:53:10.737871 2517 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 111. Preceding OpId from leader: term: 1 index: 112. (index mismatch)
I20240502 01:53:10.743173 3604 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 111. Preceding OpId from leader: term: 1 index: 112. (index mismatch)
I20240502 01:53:10.745323 4206 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 110, Time since last communication: 0.005s
I20240502 01:53:10.751670 3819 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 110, Time since last communication: 0.006s
I20240502 01:53:10.753742 3688 raft_consensus.cc:1050] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119: attempting to promote NON_VOTER 446c7f9bf31a4acc9dae8daf43330906 to VOTER
I20240502 01:53:10.755326 3688 consensus_queue.cc:237] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 1.101, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } }
I20240502 01:53:10.758086 4183 raft_consensus.cc:1013] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [term 1 LEADER]: attempt to promote peer 446c7f9bf31a4acc9dae8daf43330906: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 01:53:10.758630 2120 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240502 01:53:10.758884 2648 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240502 01:53:10.759330 3941 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20240502 01:53:10.759567 4159 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20240502 01:53:10.763906 2254 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEARNER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240502 01:53:10.764524 3941 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20240502 01:53:10.771440 3941 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [term 1 LEADER]: Committing config change with OpId 1.102: config changed from index 79 to 102, 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } }
I20240502 01:53:10.786105 3879 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 110, Time since last communication: 0.034s
I20240502 01:53:10.791244 4205 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 112, Last known committed idx: 110, Time since last communication: 0.005s
I20240502 01:53:10.794844 1990 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240502 01:53:10.795828 4159 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20240502 01:53:10.797559 1990 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 79 to 102, 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } }
I20240502 01:53:10.806687 1848 catalog_manager.cc:5484] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879 reported cstate change: config changed from index 79 to 102, 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d99e1fdda704211b3d101f892c57119" committed_config { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } }
I20240502 01:53:10.811762 2385 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 102. (index mismatch)
I20240502 01:53:10.811712 2644 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 79 to 102, 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } }
I20240502 01:53:10.812382 4159 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 102, Last known committed idx: 101, Time since last communication: 0.000s
I20240502 01:53:10.813557 2251 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 79 to 102, 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } }
I20240502 01:53:10.813822 2385 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 79 to 102, 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } }
I20240502 01:53:10.817803 2120 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.102: config changed from index 79 to 102, 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) changed from NON_VOTER to VOTER. New config: { opid_index: 102 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } }
I20240502 01:53:10.852443 2517 consensus_queue.cc:237] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 1.102, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } }
W20240502 01:53:10.854609 2322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 104467us
I20240502 01:53:10.855800 2120 raft_consensus.cc:1203] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.110->[1.111-1.112] Dedup: 1.111->[1.112-1.112]
I20240502 01:53:10.855912 2251 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240502 01:53:10.856899 2385 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
W20240502 01:53:10.857380 2321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.199:46381, user_credentials={real_user=slave}} blocked reactor thread for 107020us
I20240502 01:53:10.857213 1990 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240502 01:53:10.859184 3688 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20240502 01:53:10.859841 3688 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20240502 01:53:10.860455 4268 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20240502 01:53:10.866645 3604 raft_consensus.cc:1203] T c291b2a5e9d140f881b2bd54f8c9ed19 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.81->[1.82-1.88] Dedup: 1.83->[1.84-1.88]
I20240502 01:53:10.869638 4183 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [term 1 LEADER]: Committing config change with OpId 1.103: config changed from index 102 to 103, VOTER 9a3448a425c041258991dc60dd64ae7d (127.31.161.198) evicted. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } }
I20240502 01:53:10.870622 2120 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240502 01:53:10.871646 2385 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index 102 to 103, VOTER 9a3448a425c041258991dc60dd64ae7d (127.31.161.198) evicted. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } }
I20240502 01:53:10.872864 4159 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20240502 01:53:10.876351 1834 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e4672bc55a874f159bf10855f0698ba1 with cas_config_opid_index 102: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:10.876684 2120 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index 102 to 103, VOTER 9a3448a425c041258991dc60dd64ae7d (127.31.161.198) evicted. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } }
I20240502 01:53:10.878952 1991 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index 102 to 103, VOTER 9a3448a425c041258991dc60dd64ae7d (127.31.161.198) evicted. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } }
I20240502 01:53:10.879557 1848 catalog_manager.cc:5484] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 reported cstate change: config changed from index 102 to 103, VOTER 9a3448a425c041258991dc60dd64ae7d (127.31.161.198) evicted. New cstate: current_term: 1 leader_uuid: "4d99e1fdda704211b3d101f892c57119" committed_config { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:10.877727 2251 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index 102 to 103, VOTER 9a3448a425c041258991dc60dd64ae7d (127.31.161.198) evicted. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } }
I20240502 01:53:10.880964 3865 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Committing config change with OpId 1.112: config changed from index 86 to 112, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:10.884346 2630 tablet_service.cc:1514] Processing DeleteTablet for tablet e4672bc55a874f159bf10855f0698ba1 with delete_type TABLET_DATA_TOMBSTONED (TS 9a3448a425c041258991dc60dd64ae7d not found in new config with opid_index 103) from {username='slave'} at 127.0.0.1:52494
I20240502 01:53:10.891414 2124 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.112: config changed from index 86 to 112, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:10.896709 1848 catalog_manager.cc:5484] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea reported cstate change: config changed from index 86 to 112, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ef43ea75ecb46ee905b781c91aed95a" committed_config { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:10.897783 1991 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.112: config changed from index 86 to 112, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:10.903611 3603 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.112: config changed from index 86 to 112, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:10.890161 2383 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index 51 to 77, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
W20240502 01:53:10.924130 2319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.197:38987, user_credentials={real_user=slave}} blocked reactor thread for 173502us
I20240502 01:53:10.929628 2648 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.112: config changed from index 86 to 112, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:10.964726 2516 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.112: config changed from index 86 to 112, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 112 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:10.972592 3819 raft_consensus.cc:1050] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a: attempting to promote NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 to VOTER
I20240502 01:53:10.973527 3819 consensus_queue.cc:237] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 89, Committed index: 89, Last appended: 1.90, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } }
I20240502 01:53:10.984305 2255 raft_consensus.cc:1261] T c291b2a5e9d140f881b2bd54f8c9ed19 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20240502 01:53:10.986841 3604 raft_consensus.cc:1261] T c291b2a5e9d140f881b2bd54f8c9ed19 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 89. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20240502 01:53:10.990264 2119 raft_consensus.cc:1261] T c291b2a5e9d140f881b2bd54f8c9ed19 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20240502 01:53:10.998489 2517 raft_consensus.cc:1261] T c291b2a5e9d140f881b2bd54f8c9ed19 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 89. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20240502 01:53:11.001219 4205 consensus_queue.cc:1035] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 89, Time since last communication: 0.000s
I20240502 01:53:11.018923 1991 raft_consensus.cc:1261] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 90. Preceding OpId from leader: term: 1 index: 91. (index mismatch)
I20240502 01:53:11.021605 4205 consensus_queue.cc:1035] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 89, Time since last communication: 0.000s
I20240502 01:53:11.023170 3605 raft_consensus.cc:1203] T c291b2a5e9d140f881b2bd54f8c9ed19 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.89->[1.90-1.91] Dedup: 1.90->[1.91-1.91]
I20240502 01:53:11.027083 4265 consensus_queue.cc:1035] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 89, Time since last communication: 0.000s
I20240502 01:53:11.027501 4205 consensus_queue.cc:1035] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 89, Time since last communication: 0.000s
I20240502 01:53:11.035120 4205 consensus_queue.cc:1035] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 91, Last known committed idx: 89, Time since last communication: 0.000s
I20240502 01:53:11.045408 4266 raft_consensus.cc:2932] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Committing config change with OpId 1.91: config changed from index 82 to 91, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.052989 3605 raft_consensus.cc:2932] T c291b2a5e9d140f881b2bd54f8c9ed19 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 82 to 91, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.053186 1988 raft_consensus.cc:2932] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 82 to 91, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.058472 1841 catalog_manager.cc:5484] T c291b2a5e9d140f881b2bd54f8c9ed19 P 163649b5fbb140f18dccdf75f608b0f9 reported cstate change: config changed from index 82 to 91, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ef43ea75ecb46ee905b781c91aed95a" committed_config { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.060890 2648 consensus_queue.cc:237] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 1.79, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } }
I20240502 01:53:11.075438 2385 consensus_queue.cc:237] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 113, Committed index: 113, Last appended: 1.114, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } }
I20240502 01:53:11.077972 2649 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 114. Preceding OpId from leader: term: 1 index: 115. (index mismatch)
I20240502 01:53:11.080619 4205 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 113, Time since last communication: 0.000s
I20240502 01:53:11.085065 2119 raft_consensus.cc:2932] T c291b2a5e9d140f881b2bd54f8c9ed19 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 82 to 91, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.098883 2254 raft_consensus.cc:2932] T c291b2a5e9d140f881b2bd54f8c9ed19 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 82 to 91, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
W20240502 01:53:11.109269 2583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.195:39353, user_credentials={real_user=slave}} blocked reactor thread for 47504.6us
I20240502 01:53:11.117488 2120 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240502 01:53:11.118196 3933 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20240502 01:53:11.120731 3605 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240502 01:53:11.121454 3933 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20240502 01:53:11.124606 4249 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [term 1 LEADER]: Committing config change with OpId 1.80: config changed from index 77 to 80, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.125216 3604 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index 77 to 80, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.127856 2254 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240502 01:53:11.121456 4273 tablet_replica.cc:330] T e4672bc55a874f159bf10855f0698ba1 P 9a3448a425c041258991dc60dd64ae7d: stopping tablet replica
I20240502 01:53:11.128892 4273 raft_consensus.cc:2229] T e4672bc55a874f159bf10855f0698ba1 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:11.129220 3798 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20240502 01:53:11.130059 1834 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e567c877188549fc965637e89bf437ae with cas_config_opid_index 77: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:11.130722 2123 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index 77 to 80, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.131873 1848 catalog_manager.cc:5484] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9 reported cstate change: config changed from index 77 to 80, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "9a3448a425c041258991dc60dd64ae7d" committed_config { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.138926 1971 tablet_service.cc:1514] Processing DeleteTablet for tablet e567c877188549fc965637e89bf437ae with delete_type TABLET_DATA_TOMBSTONED (TS 58dae5dc94444d5e92e6907c5b637879 not found in new config with opid_index 80) from {username='slave'} at 127.0.0.1:42016
I20240502 01:53:11.141039 2254 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index 77 to 80, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.145438 4273 raft_consensus.cc:2258] T e4672bc55a874f159bf10855f0698ba1 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:11.151262 4273 ts_tablet_manager.cc:1875] T e4672bc55a874f159bf10855f0698ba1 P 9a3448a425c041258991dc60dd64ae7d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:11.153586 2387 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240502 01:53:11.162031 3604 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 114. Preceding OpId from leader: term: 1 index: 115. (index mismatch)
I20240502 01:53:11.164362 2123 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 114. Preceding OpId from leader: term: 1 index: 115. (index mismatch)
I20240502 01:53:11.168073 1988 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 114. Preceding OpId from leader: term: 1 index: 115. (index mismatch)
I20240502 01:53:11.174481 4205 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 113, Time since last communication: 0.013s
I20240502 01:53:11.159610 4071 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20240502 01:53:11.178222 4205 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 113, Time since last communication: 0.000s
I20240502 01:53:11.180532 4266 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Committing config change with OpId 1.115: config changed from index 112 to 115, VOTER 4d99e1fdda704211b3d101f892c57119 (127.31.161.197) evicted. New config: { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.181303 3604 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.115: config changed from index 112 to 115, VOTER 4d99e1fdda704211b3d101f892c57119 (127.31.161.197) evicted. New config: { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.182898 1988 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.115: config changed from index 112 to 115, VOTER 4d99e1fdda704211b3d101f892c57119 (127.31.161.197) evicted. New config: { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.190533 1847 catalog_manager.cc:5484] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9 reported cstate change: config changed from index 112 to 115, VOTER 4d99e1fdda704211b3d101f892c57119 (127.31.161.197) evicted. New cstate: current_term: 1 leader_uuid: "7ef43ea75ecb46ee905b781c91aed95a" committed_config { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.192744 2648 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.115: config changed from index 112 to 115, VOTER 4d99e1fdda704211b3d101f892c57119 (127.31.161.197) evicted. New config: { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.194900 2496 tablet_service.cc:1514] Processing DeleteTablet for tablet 4220d1a63bd74ad0a5cac5a2f1bb988c with delete_type TABLET_DATA_TOMBSTONED (TS 4d99e1fdda704211b3d101f892c57119 not found in new config with opid_index 115) from {username='slave'} at 127.0.0.1:48022
I20240502 01:53:11.207197 2384 consensus_queue.cc:237] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 1.92, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } }
I20240502 01:53:11.208556 2516 raft_consensus.cc:2932] T c291b2a5e9d140f881b2bd54f8c9ed19 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.91: config changed from index 82 to 91, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 91 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.211197 1835 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4220d1a63bd74ad0a5cac5a2f1bb988c with cas_config_opid_index 112: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:11.212725 2120 raft_consensus.cc:1261] T c291b2a5e9d140f881b2bd54f8c9ed19 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 91. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240502 01:53:11.212944 4266 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 115, Last known committed idx: 113, Time since last communication: 0.001s
I20240502 01:53:11.213929 2384 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index 77 to 80, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.222203 3604 raft_consensus.cc:1261] T c291b2a5e9d140f881b2bd54f8c9ed19 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240502 01:53:11.223147 4205 consensus_queue.cc:1035] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 91, Time since last communication: 0.000s
I20240502 01:53:11.234054 4285 tablet_replica.cc:330] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119: stopping tablet replica
I20240502 01:53:11.234900 4285 raft_consensus.cc:2229] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:11.235296 4273 ts_tablet_manager.cc:1888] T e4672bc55a874f159bf10855f0698ba1 P 9a3448a425c041258991dc60dd64ae7d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.102
I20240502 01:53:11.235474 4273 log.cc:1196] T e4672bc55a874f159bf10855f0698ba1 P 9a3448a425c041258991dc60dd64ae7d: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal/wals/e4672bc55a874f159bf10855f0698ba1
I20240502 01:53:11.238591 1834 catalog_manager.cc:4830] TS 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585): tablet e4672bc55a874f159bf10855f0698ba1 (table rebalance_test_table_3 [id=3402138c5f114264ac2eb0f198dbe6b3]) successfully deleted
I20240502 01:53:11.241271 4205 consensus_queue.cc:1035] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 91, Time since last communication: 0.014s
I20240502 01:53:11.255079 2255 raft_consensus.cc:1261] T c291b2a5e9d140f881b2bd54f8c9ed19 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240502 01:53:11.265759 4282 tablet_replica.cc:330] T e567c877188549fc965637e89bf437ae P 58dae5dc94444d5e92e6907c5b637879: stopping tablet replica
I20240502 01:53:11.266396 4282 raft_consensus.cc:2229] T e567c877188549fc965637e89bf437ae P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:11.267532 4282 raft_consensus.cc:2258] T e567c877188549fc965637e89bf437ae P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:11.269845 3819 raft_consensus.cc:2932] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Committing config change with OpId 1.93: config changed from index 91 to 93, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.278192 2120 raft_consensus.cc:2932] T c291b2a5e9d140f881b2bd54f8c9ed19 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 91 to 93, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.279891 4282 ts_tablet_manager.cc:1875] T e567c877188549fc965637e89bf437ae P 58dae5dc94444d5e92e6907c5b637879: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:11.287580 1847 catalog_manager.cc:5484] T c291b2a5e9d140f881b2bd54f8c9ed19 P 269c9061e1a1435e92f79f0b9486dfea reported cstate change: config changed from index 91 to 93, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "7ef43ea75ecb46ee905b781c91aed95a" committed_config { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.288447 1835 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c291b2a5e9d140f881b2bd54f8c9ed19 with cas_config_opid_index 91: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:11.302060 1971 tablet_service.cc:1514] Processing DeleteTablet for tablet c291b2a5e9d140f881b2bd54f8c9ed19 with delete_type TABLET_DATA_TOMBSTONED (TS 58dae5dc94444d5e92e6907c5b637879 not found in new config with opid_index 93) from {username='slave'} at 127.0.0.1:42016
I20240502 01:53:11.309824 4282 ts_tablet_manager.cc:1888] T e567c877188549fc965637e89bf437ae P 58dae5dc94444d5e92e6907c5b637879: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.79
I20240502 01:53:11.308676 3604 raft_consensus.cc:2932] T c291b2a5e9d140f881b2bd54f8c9ed19 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 91 to 93, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.310143 4282 log.cc:1196] T e567c877188549fc965637e89bf437ae P 58dae5dc94444d5e92e6907c5b637879: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/e567c877188549fc965637e89bf437ae
I20240502 01:53:11.311127 1836 catalog_manager.cc:4830] TS 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): tablet e567c877188549fc965637e89bf437ae (table rebalance_test_table_4 [id=2cd052d4ae9b43129f79f513560af30d]) successfully deleted
I20240502 01:53:11.316972 4282 tablet_replica.cc:330] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879: stopping tablet replica
I20240502 01:53:11.303766 2124 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.115: config changed from index 112 to 115, VOTER 4d99e1fdda704211b3d101f892c57119 (127.31.161.197) evicted. New config: { opid_index: 115 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.317432 4282 raft_consensus.cc:2229] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:11.317765 4282 pending_rounds.cc:62] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879: Trying to abort 1 pending ops.
I20240502 01:53:11.317932 4282 pending_rounds.cc:68] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879: Aborting op as it isn't in flight: id { term: 1 index: 92 } timestamp: 7023062184371605504 op_type: WRITE_OP write_request { tablet_id: "c291b2a5e9d140f881b2bd54f8c9ed19" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\332P8,\331\215b\037\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""03b8c61eafadfc9c" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023062184228872192 authz_token { token_data: "\010\247\346\313\261\006\"3\n\005slave\022*\n 9b1fbaeec6af4ae5a31888af58780907\020\001\030\001 \001(\001" signature: "<redacted>""u\257\264\302<\332\374\373\200\246 |\314\260\t\214\006$\254\252\260\267\326]\245\301>w\207\211d\3041\200\375\203i\217\321\202\022\364\244UM\305\001$\3770zQ]x=\033?\334\240r\236\024\007\270" signing_key_seq_num: 0 } } request_id { client_id: "182f50bc068d45019f86298963bd14f1" seq_no: 316 first_incomplete_seq_no: 316 attempt_no: 0 }
I20240502 01:53:11.318768 4282 raft_consensus.cc:2258] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:11.321591 4282 ts_tablet_manager.cc:1875] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:11.322448 4285 pending_rounds.cc:62] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119: Trying to abort 1 pending ops.
I20240502 01:53:11.324748 4285 pending_rounds.cc:68] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119: Aborting op as it isn't in flight: id { term: 1 index: 114 } timestamp: 7023062184080183296 op_type: WRITE_OP write_request { tablet_id: "4220d1a63bd74ad0a5cac5a2f1bb988c" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000)I\2276\317ps\352\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""d4a3587f600e08dd" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023062183988387840 authz_token { token_data: "\010\247\346\313\261\006\"3\n\005slave\022*\n 3402138c5f114264ac2eb0f198dbe6b3\020\001\030\001 \001(\001" signature: "<redacted>""\273$\022h<\031\363\204F\000\022Q\225\260]\347=\360\346`E\037\261\226\204\017\221\373D\031G\357\352\252\252]\336\321\270\207\314YJ\225@\214h/.\242g\215\225^\021\3360`\324/\"\254a}" signing_key_seq_num: 0 } } request_id { client_id: "e04699195f894f4b8845d7aca532619e" seq_no: 300 first_incomplete_seq_no: 300 attempt_no: 0 }
I20240502 01:53:11.336694 4282 ts_tablet_manager.cc:1888] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.92
I20240502 01:53:11.337219 4282 log.cc:1196] T c291b2a5e9d140f881b2bd54f8c9ed19 P 58dae5dc94444d5e92e6907c5b637879: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/c291b2a5e9d140f881b2bd54f8c9ed19
I20240502 01:53:11.337733 4267 consensus_queue.cc:1035] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 91, Time since last communication: 0.000s
I20240502 01:53:11.338483 1836 catalog_manager.cc:4830] TS 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): tablet c291b2a5e9d140f881b2bd54f8c9ed19 (table rebalance_test_table_0 [id=9b1fbaeec6af4ae5a31888af58780907]) successfully deleted
I20240502 01:53:11.325829 2515 raft_consensus.cc:1261] T c291b2a5e9d140f881b2bd54f8c9ed19 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240502 01:53:11.384882 4285 raft_consensus.cc:2258] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:11.406327 4266 consensus_queue.cc:1035] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 91, Time since last communication: 0.000s
I20240502 01:53:11.412930 2255 raft_consensus.cc:2932] T c291b2a5e9d140f881b2bd54f8c9ed19 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 91 to 93, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.444073 4137 tablet_bootstrap.cc:492] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=81 overwritten=0 applied=80 ignored=0} inserts{seen=752 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:53:11.444551 4137 tablet_bootstrap.cc:492] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap complete.
I20240502 01:53:11.444866 4137 ts_tablet_manager.cc:1367] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9: Time spent bootstrapping tablet: real 1.408s user 0.127s sys 0.016s
I20240502 01:53:11.445624 4137 raft_consensus.cc:348] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:11.451936 4285 ts_tablet_manager.cc:1875] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 01:53:11.460054 2319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.198:37585, user_credentials={real_user=slave}} blocked reactor thread for 52396.1us
I20240502 01:53:11.464767 4137 raft_consensus.cc:729] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 163649b5fbb140f18dccdf75f608b0f9, State: Initialized, Role: LEARNER
I20240502 01:53:11.465132 4137 consensus_queue.cc:260] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 80, Last appended: 1.81, Last appended by leader: 81, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:11.466953 4137 ts_tablet_manager.cc:1398] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20240502 01:53:11.477413 4285 ts_tablet_manager.cc:1888] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.114
I20240502 01:53:11.477646 4285 log.cc:1196] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 4d99e1fdda704211b3d101f892c57119: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal/wals/4220d1a63bd74ad0a5cac5a2f1bb988c
I20240502 01:53:11.478633 1834 catalog_manager.cc:4830] TS 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987): tablet 4220d1a63bd74ad0a5cac5a2f1bb988c (table rebalance_test_table_3 [id=3402138c5f114264ac2eb0f198dbe6b3]) successfully deleted
I20240502 01:53:11.513144 2515 raft_consensus.cc:2932] T c291b2a5e9d140f881b2bd54f8c9ed19 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 91 to 93, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:11.525285 2275 tablet_copy_service.cc:342] P 446c7f9bf31a4acc9dae8daf43330906: Request end of tablet copy session 163649b5fbb140f18dccdf75f608b0f9-f4a07f0fe1964f958ad7ca4a63d054a6 received from {username='slave'} at 127.31.161.199:59365
I20240502 01:53:11.525460 2275 tablet_copy_service.cc:433] P 446c7f9bf31a4acc9dae8daf43330906: ending tablet copy session 163649b5fbb140f18dccdf75f608b0f9-f4a07f0fe1964f958ad7ca4a63d054a6 on tablet f4a07f0fe1964f958ad7ca4a63d054a6 with peer 163649b5fbb140f18dccdf75f608b0f9
I20240502 01:53:11.526247 3603 raft_consensus.cc:1203] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.64->[1.65-1.101] Dedup: 1.81->[1.82-1.101]
W20240502 01:53:11.787087 2321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.199:46381, user_credentials={real_user=slave}} blocked reactor thread for 56705.6us
I20240502 01:53:11.898658 4020 raft_consensus.cc:1050] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906: attempting to promote NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 to VOTER
I20240502 01:53:11.900059 4020 consensus_queue.cc:237] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 1.103, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } }
I20240502 01:53:11.905809 2644 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240502 01:53:11.906443 4236 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240502 01:53:11.907536 2384 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240502 01:53:11.920523 4236 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240502 01:53:11.940433 2517 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
W20240502 01:53:11.940668 2440 kernel_stack_watchdog.cc:198] Thread 2453 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 112ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f5c6f10b980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7f5c6bd79d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7f5c6bd9a7ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f5c6bd963ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f5c6bd9484f kudu::ThreadPoolToken::Submit() at ??:0
@ 0x7f5c774c79e8 kudu::consensus::Peer::ProcessResponse() at ??:0
@ 0x7f5c774ce061 kudu::consensus::Peer::SendNextRequest()::$_3::operator()() at ??:0
@ 0x7f5c774cde6e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f5c729cb8cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f5c729cb4ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f5c729b74e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
I20240502 01:53:11.949180 3605 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240502 01:53:11.951810 1988 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240502 01:53:11.954939 4236 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 102, Time since last communication: 0.000s
I20240502 01:53:11.980036 4236 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240502 01:53:11.982308 4236 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
W20240502 01:53:11.999372 2453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 269711us
I20240502 01:53:11.999720 4170 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEADER]: Committing config change with OpId 1.104: config changed from index 65 to 104, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:12.006868 2649 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 65 to 104, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:12.008548 1847 catalog_manager.cc:5484] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 reported cstate change: config changed from index 65 to 104, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "446c7f9bf31a4acc9dae8daf43330906" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:12.022961 2517 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 65 to 104, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:12.024556 1991 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 65 to 104, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:12.033887 3605 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 65 to 104, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:12.010161 2384 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 65 to 104, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:12.233505 2249 consensus_queue.cc:237] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } }
I20240502 01:53:12.236264 1991 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch)
I20240502 01:53:12.236259 2644 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch)
I20240502 01:53:12.238466 4283 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20240502 01:53:12.239135 4283 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20240502 01:53:12.242053 4283 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index 104 to 106, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
W20240502 01:53:12.245429 1925 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.198:37585, user_credentials={real_user=slave}} blocked reactor thread for 58365.9us
I20240502 01:53:12.246897 3605 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch)
I20240502 01:53:12.247766 4236 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20240502 01:53:12.249352 1834 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f4a07f0fe1964f958ad7ca4a63d054a6 with cas_config_opid_index 104: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:12.249771 3602 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 104 to 106, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:12.250033 1988 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 104 to 106, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:12.251431 1844 catalog_manager.cc:5484] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 reported cstate change: config changed from index 104 to 106, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "446c7f9bf31a4acc9dae8daf43330906" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:12.253909 2644 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 104 to 106, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:12.257215 2514 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch)
I20240502 01:53:12.259254 2366 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a07f0fe1964f958ad7ca4a63d054a6 with delete_type TABLET_DATA_TOMBSTONED (TS 7ef43ea75ecb46ee905b781c91aed95a not found in new config with opid_index 106) from {username='slave'} at 127.0.0.1:54148
I20240502 01:53:12.264829 4170 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20240502 01:53:12.274681 2514 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 104 to 106, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:12.283111 4337 tablet_replica.cc:330] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a: stopping tablet replica
I20240502 01:53:12.283366 4337 raft_consensus.cc:2229] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:12.284550 4337 pending_rounds.cc:62] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a: Trying to abort 1 pending ops.
I20240502 01:53:12.284675 4337 pending_rounds.cc:68] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a: Aborting op as it isn't in flight: id { term: 1 index: 105 } timestamp: 7023062187972751360 op_type: WRITE_OP write_request { tablet_id: "f4a07f0fe1964f958ad7ca4a63d054a6" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\003\300@HYvxl\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""3a7116f62d351ce7" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023062187691831296 authz_token { token_data: "\010\247\346\313\261\006\"3\n\005slave\022*\n 2cd052d4ae9b43129f79f513560af30d\020\001\030\001 \001(\001" signature: "<redacted>""\330\205\313X9\350-\224W\024Xx/\177\362%T\031\337~\347\213\210\353\270H\247s\031\212h\006\327\352\212\260\320\320\277f\302\034\034\322\021\327[\323\221\322OI\340\312S\335m#\376\021\370\230\313\227" signing_key_seq_num: 0 } } request_id { client_id: "8ad82ffc1c144b76a0f03413ca2d08fd" seq_no: 275 first_incomplete_seq_no: 275 attempt_no: 0 }
I20240502 01:53:12.285552 4337 raft_consensus.cc:2258] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:12.288484 4337 ts_tablet_manager.cc:1875] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:12.367348 4337 ts_tablet_manager.cc:1888] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.105
I20240502 01:53:12.367519 4337 log.cc:1196] T f4a07f0fe1964f958ad7ca4a63d054a6 P 7ef43ea75ecb46ee905b781c91aed95a: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/f4a07f0fe1964f958ad7ca4a63d054a6
I20240502 01:53:12.368360 1835 catalog_manager.cc:4830] TS 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225): tablet f4a07f0fe1964f958ad7ca4a63d054a6 (table rebalance_test_table_4 [id=2cd052d4ae9b43129f79f513560af30d]) successfully deleted
I20240502 01:53:12.459316 2255 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "341e0200ea654dcfb91133135b458a16"
dest_uuid: "446c7f9bf31a4acc9dae8daf43330906"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49756
I20240502 01:53:12.459606 2255 raft_consensus.cc:595] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEADER]: Received request to transfer leadership
W20240502 01:53:12.510670 2213 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45474: Service unavailable: leader transfer in progress
W20240502 01:53:12.516805 2213 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45474: Service unavailable: leader transfer in progress
W20240502 01:53:12.524363 2213 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45474: Service unavailable: leader transfer in progress
W20240502 01:53:12.542975 2213 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45474: Service unavailable: leader transfer in progress
W20240502 01:53:12.578414 2213 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45474: Service unavailable: leader transfer in progress
I20240502 01:53:12.581420 4283 raft_consensus.cc:979] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906: : Instructing follower 4d99e1fdda704211b3d101f892c57119 to start an election
I20240502 01:53:12.581626 4283 raft_consensus.cc:1067] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEADER]: Signalling peer 4d99e1fdda704211b3d101f892c57119 to start an election
I20240502 01:53:12.583371 2514 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "341e0200ea654dcfb91133135b458a16"
dest_uuid: "4d99e1fdda704211b3d101f892c57119"
from {username='slave'} at 127.31.161.195:46969
I20240502 01:53:12.583669 2514 raft_consensus.cc:482] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:12.583807 2514 raft_consensus.cc:3037] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Advancing to term 2
W20240502 01:53:12.590202 2213 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45474: Service unavailable: leader transfer in progress
I20240502 01:53:12.605250 2514 raft_consensus.cc:504] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: false } }
I20240502 01:53:12.616161 2252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341e0200ea654dcfb91133135b458a16" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 2 candidate_status { last_received { term: 1 index: 122 } } ignore_live_leader: true dest_uuid: "446c7f9bf31a4acc9dae8daf43330906"
I20240502 01:53:12.616412 2252 raft_consensus.cc:3032] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:12.616536 2252 raft_consensus.cc:729] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 446c7f9bf31a4acc9dae8daf43330906, State: Running, Role: LEADER
I20240502 01:53:12.616902 2252 consensus_queue.cc:260] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 1.122, Last appended by leader: 122, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: false } }
I20240502 01:53:12.617552 2252 raft_consensus.cc:3037] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:12.620312 2252 raft_consensus.cc:2454] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 2.
W20240502 01:53:12.625206 2213 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45474: Illegal state: replica 446c7f9bf31a4acc9dae8daf43330906 is not leader of this config: current role FOLLOWER
I20240502 01:53:12.633546 2644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341e0200ea654dcfb91133135b458a16" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 2 candidate_status { last_received { term: 1 index: 122 } } ignore_live_leader: true dest_uuid: "9a3448a425c041258991dc60dd64ae7d"
I20240502 01:53:12.633777 2644 raft_consensus.cc:3037] T 341e0200ea654dcfb91133135b458a16 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:12.636886 2387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341e0200ea654dcfb91133135b458a16" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 2 candidate_status { last_received { term: 1 index: 122 } } ignore_live_leader: true dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a"
I20240502 01:53:12.637037 2644 raft_consensus.cc:2454] T 341e0200ea654dcfb91133135b458a16 P 9a3448a425c041258991dc60dd64ae7d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 2.
I20240502 01:53:12.637179 2387 raft_consensus.cc:3037] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:12.658155 2387 raft_consensus.cc:2454] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 2.
I20240502 01:53:12.658675 2452 leader_election.cc:302] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 6 voters: 4 yes votes; 0 no votes. yes voters: 446c7f9bf31a4acc9dae8daf43330906, 4d99e1fdda704211b3d101f892c57119, 7ef43ea75ecb46ee905b781c91aed95a, 9a3448a425c041258991dc60dd64ae7d; no voters:
I20240502 01:53:12.659034 3941 raft_consensus.cc:2781] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:12.659181 3941 raft_consensus.cc:686] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [term 2 LEADER]: Becoming Leader. State: Replica: 4d99e1fdda704211b3d101f892c57119, State: Running, Role: LEADER
I20240502 01:53:12.659499 3941 consensus_queue.cc:237] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 1.122, Last appended by leader: 122, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: false } }
I20240502 01:53:12.668406 1988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341e0200ea654dcfb91133135b458a16" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 2 candidate_status { last_received { term: 1 index: 122 } } ignore_live_leader: true dest_uuid: "58dae5dc94444d5e92e6907c5b637879"
I20240502 01:53:12.668648 1988 raft_consensus.cc:3037] T 341e0200ea654dcfb91133135b458a16 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:12.671420 1988 raft_consensus.cc:2454] T 341e0200ea654dcfb91133135b458a16 P 58dae5dc94444d5e92e6907c5b637879 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 2.
I20240502 01:53:12.710599 2385 raft_consensus.cc:1226] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a [term 2 FOLLOWER]: Rejecting Update request from peer 446c7f9bf31a4acc9dae8daf43330906 for earlier term 1. Current term is 2. Ops: []
I20240502 01:53:12.711114 2120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "341e0200ea654dcfb91133135b458a16" candidate_uuid: "4d99e1fdda704211b3d101f892c57119" candidate_term: 2 candidate_status { last_received { term: 1 index: 122 } } ignore_live_leader: true dest_uuid: "269c9061e1a1435e92f79f0b9486dfea"
I20240502 01:53:12.711374 2120 raft_consensus.cc:3037] T 341e0200ea654dcfb91133135b458a16 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:12.714928 2120 raft_consensus.cc:2454] T 341e0200ea654dcfb91133135b458a16 P 269c9061e1a1435e92f79f0b9486dfea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d99e1fdda704211b3d101f892c57119 in term 2.
I20240502 01:53:12.730993 2514 leader_election.cc:288] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [CANDIDATE]: Term 2 election: Requested vote from peers 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621)
I20240502 01:53:12.765612 1844 catalog_manager.cc:5484] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 reported cstate change: term changed from 1 to 2, leader changed from 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) to 4d99e1fdda704211b3d101f892c57119 (127.31.161.197). New cstate: current_term: 2 leader_uuid: "4d99e1fdda704211b3d101f892c57119" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:12.787516 2121 raft_consensus.cc:1261] T 341e0200ea654dcfb91133135b458a16 P 269c9061e1a1435e92f79f0b9486dfea [term 2 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 2 index: 124. (index mismatch)
I20240502 01:53:12.788091 3941 consensus_queue.cc:1035] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20240502 01:53:12.798307 2384 raft_consensus.cc:1261] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a [term 2 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 2 index: 124. (index mismatch)
I20240502 01:53:12.799237 2649 raft_consensus.cc:1261] T 341e0200ea654dcfb91133135b458a16 P 9a3448a425c041258991dc60dd64ae7d [term 2 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 2 index: 124. (index mismatch)
I20240502 01:53:12.799760 3941 consensus_queue.cc:1035] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 121, Time since last communication: 0.000s
I20240502 01:53:12.800770 3941 consensus_queue.cc:1035] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20240502 01:53:12.800549 1988 raft_consensus.cc:1261] T 341e0200ea654dcfb91133135b458a16 P 58dae5dc94444d5e92e6907c5b637879 [term 2 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 2 index: 124. (index mismatch)
I20240502 01:53:12.803244 4305 consensus_queue.cc:1035] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20240502 01:53:12.805352 2249 raft_consensus.cc:1261] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [term 2 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 2 index: 124. (index mismatch)
I20240502 01:53:12.805903 3941 consensus_queue.cc:1035] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 122, Time since last communication: 0.000s
I20240502 01:53:12.813036 4302 mvcc.cc:205] Tried to move back new op lower bound from 7023062191217729536 to 7023062190739283968. Current Snapshot: MvccSnapshot[applied={T|T < 7023062191217729536}]
I20240502 01:53:12.813418 4109 mvcc.cc:205] Tried to move back new op lower bound from 7023062191217729536 to 7023062190739283968. Current Snapshot: MvccSnapshot[applied={T|T < 7023062191217729536 or (T in {7023062191217729536})}]
W20240502 01:53:12.934806 2322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 50774.7us
I20240502 01:53:12.950336 2251 consensus_queue.cc:237] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 104, Committed index: 104, Last appended: 1.104, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:12.953670 2514 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240502 01:53:12.954236 2387 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240502 01:53:12.954269 4020 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20240502 01:53:12.954725 4020 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20240502 01:53:12.959753 1988 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240502 01:53:12.967002 2644 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 104. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240502 01:53:12.968987 4170 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.000s
I20240502 01:53:12.969947 4020 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEADER]: Committing config change with OpId 1.105: config changed from index -1 to 105, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:12.981959 2516 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index -1 to 105, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:12.988739 2385 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index -1 to 105, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:12.995648 2644 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index -1 to 105, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:13.001381 4020 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 105, Last known committed idx: 104, Time since last communication: 0.001s
W20240502 01:53:13.002722 2189 consensus_peers.cc:485] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 -> Peer 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381): Couldn't send request to peer 163649b5fbb140f18dccdf75f608b0f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d592edf3298942cd93739aa846f867a4. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:13.014321 1988 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.105: config changed from index -1 to 105, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
W20240502 01:53:13.023156 2585 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.199:46381, user_credentials={real_user=slave}} blocked reactor thread for 234649us
I20240502 01:53:13.093989 1847 catalog_manager.cc:5484] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 reported cstate change: config changed from index -1 to 105, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New cstate: current_term: 1 leader_uuid: "446c7f9bf31a4acc9dae8daf43330906" committed_config { opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:53:13.153301 2189 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 75561us
I20240502 01:53:13.244452 2514 consensus_queue.cc:237] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 130, Committed index: 130, Last appended: 2.130, Last appended by leader: 122, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: false } }
I20240502 01:53:13.248919 1987 raft_consensus.cc:1261] T 341e0200ea654dcfb91133135b458a16 P 58dae5dc94444d5e92e6907c5b637879 [term 2 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 2 index: 130. Preceding OpId from leader: term: 2 index: 131. (index mismatch)
I20240502 01:53:13.249969 2122 raft_consensus.cc:1261] T 341e0200ea654dcfb91133135b458a16 P 269c9061e1a1435e92f79f0b9486dfea [term 2 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 2 index: 130. Preceding OpId from leader: term: 2 index: 131. (index mismatch)
I20240502 01:53:13.250921 4268 consensus_queue.cc:1035] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20240502 01:53:13.252120 4268 consensus_queue.cc:1035] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20240502 01:53:13.267022 4268 raft_consensus.cc:2932] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [term 2 LEADER]: Committing config change with OpId 2.131: config changed from index 108 to 131, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: false } } }
I20240502 01:53:13.271311 1834 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 341e0200ea654dcfb91133135b458a16 with cas_config_opid_index 108: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:13.330941 2385 raft_consensus.cc:1261] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a [term 2 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 2 index: 130. Preceding OpId from leader: term: 2 index: 131. (index mismatch)
I20240502 01:53:13.293030 2644 raft_consensus.cc:1261] T 341e0200ea654dcfb91133135b458a16 P 9a3448a425c041258991dc60dd64ae7d [term 2 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 2 index: 130. Preceding OpId from leader: term: 2 index: 131. (index mismatch)
I20240502 01:53:13.273583 1991 raft_consensus.cc:2932] T 341e0200ea654dcfb91133135b458a16 P 58dae5dc94444d5e92e6907c5b637879 [term 2 FOLLOWER]: Committing config change with OpId 2.131: config changed from index 108 to 131, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: false } } }
I20240502 01:53:13.276172 1841 catalog_manager.cc:5484] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 reported cstate change: config changed from index 108 to 131, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New cstate: current_term: 2 leader_uuid: "4d99e1fdda704211b3d101f892c57119" committed_config { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:13.268404 2121 raft_consensus.cc:2932] T 341e0200ea654dcfb91133135b458a16 P 269c9061e1a1435e92f79f0b9486dfea [term 2 FOLLOWER]: Committing config change with OpId 2.131: config changed from index 108 to 131, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: false } } }
I20240502 01:53:13.457331 4268 consensus_queue.cc:1035] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 129, Time since last communication: 0.000s
I20240502 01:53:13.460059 2385 raft_consensus.cc:2932] T 341e0200ea654dcfb91133135b458a16 P 7ef43ea75ecb46ee905b781c91aed95a [term 2 FOLLOWER]: Committing config change with OpId 2.131: config changed from index 108 to 131, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: false } } }
I20240502 01:53:13.461958 4349 consensus_queue.cc:1035] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20240502 01:53:13.462452 2234 tablet_service.cc:1514] Processing DeleteTablet for tablet 341e0200ea654dcfb91133135b458a16 with delete_type TABLET_DATA_TOMBSTONED (TS 446c7f9bf31a4acc9dae8daf43330906 not found in new config with opid_index 131) from {username='slave'} at 127.0.0.1:45438
I20240502 01:53:13.467294 2644 raft_consensus.cc:2932] T 341e0200ea654dcfb91133135b458a16 P 9a3448a425c041258991dc60dd64ae7d [term 2 FOLLOWER]: Committing config change with OpId 2.131: config changed from index 108 to 131, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { promote: false } } }
I20240502 01:53:13.500332 4387 tablet_replica.cc:330] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906: stopping tablet replica
I20240502 01:53:13.570513 4387 raft_consensus.cc:2229] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:13.621043 4387 pending_rounds.cc:62] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906: Trying to abort 1 pending ops.
I20240502 01:53:13.621353 4387 pending_rounds.cc:68] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906: Aborting op as it isn't in flight: id { term: 2 index: 130 } timestamp: 7023062193017155584 op_type: WRITE_OP write_request { tablet_id: "341e0200ea654dcfb91133135b458a16" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\371\311f\006t\035\204\267\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""4c28b82a0aa5a745" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023062192448856064 authz_token { token_data: "\010\247\346\313\261\006\"3\n\005slave\022*\n db056d9436d84c2e8d32bb9d34b3c3a1\020\001\030\001 \001(\001" signature: "<redacted>""\336M\330\347\221\247O\354>\377\304\201\014\265\342\225N<\256\315\346i@o\026{\347\302@\207z\353\373~\246\032\332\366B\2608/\022\360\374\3462?\215\"\274_Z\006\315\"\251\276T\206\222\204\277\254" signing_key_seq_num: 0 } } request_id { client_id: "0aeb65c02de34f04a7badf1447d20826" seq_no: 315 first_incomplete_seq_no: 315 attempt_no: 0 }
I20240502 01:53:13.683809 4387 raft_consensus.cc:2258] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240502 01:53:13.794989 2319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.197:38987, user_credentials={real_user=slave}} blocked reactor thread for 48306.4us
I20240502 01:53:13.813793 4387 ts_tablet_manager.cc:1875] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:13.816323 4392 ts_tablet_manager.cc:902] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9: Initiating tablet copy from peer 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353)
I20240502 01:53:13.875456 4387 ts_tablet_manager.cc:1888] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.130
I20240502 01:53:13.877921 4387 log.cc:1196] T 341e0200ea654dcfb91133135b458a16 P 446c7f9bf31a4acc9dae8daf43330906: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/wals/341e0200ea654dcfb91133135b458a16
I20240502 01:53:13.881565 1834 catalog_manager.cc:4830] TS 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353): tablet 341e0200ea654dcfb91133135b458a16 (table rebalance_test_table_1 [id=db056d9436d84c2e8d32bb9d34b3c3a1]) successfully deleted
I20240502 01:53:13.891147 4392 tablet_copy_client.cc:308] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:39353
I20240502 01:53:13.892375 2275 tablet_copy_service.cc:140] P 446c7f9bf31a4acc9dae8daf43330906: Received BeginTabletCopySession request for tablet d592edf3298942cd93739aa846f867a4 from peer 163649b5fbb140f18dccdf75f608b0f9 ({username='slave'} at 127.31.161.199:59365)
I20240502 01:53:13.892544 2275 tablet_copy_service.cc:161] P 446c7f9bf31a4acc9dae8daf43330906: Beginning new tablet copy session on tablet d592edf3298942cd93739aa846f867a4 from peer 163649b5fbb140f18dccdf75f608b0f9 at {username='slave'} at 127.31.161.199:59365: session id = 163649b5fbb140f18dccdf75f608b0f9-d592edf3298942cd93739aa846f867a4
W20240502 01:53:13.903489 1926 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.196:39225, user_credentials={real_user=slave}} blocked reactor thread for 78311.6us
W20240502 01:53:13.910328 1925 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.195:39353, user_credentials={real_user=slave}} blocked reactor thread for 102061us
I20240502 01:53:13.936995 2275 tablet_copy_source_session.cc:205] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:13.938907 4392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d592edf3298942cd93739aa846f867a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:13.998497 4392 tablet_copy_client.cc:783] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:14.002492 4392 tablet_copy_client.cc:647] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:14.006111 4392 tablet_copy_client.cc:523] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:14.011304 4392 tablet_bootstrap.cc:492] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap starting.
W20240502 01:53:14.242792 2322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 51318.8us
I20240502 01:53:14.257527 4392 tablet_bootstrap.cc:492] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=112 overwritten=0 applied=111 ignored=0} inserts{seen=783 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 01:53:14.258019 4392 tablet_bootstrap.cc:492] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap complete.
I20240502 01:53:14.258329 4392 ts_tablet_manager.cc:1367] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9: Time spent bootstrapping tablet: real 0.247s user 0.141s sys 0.031s
I20240502 01:53:14.259111 4392 raft_consensus.cc:348] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:14.260313 4392 raft_consensus.cc:729] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 163649b5fbb140f18dccdf75f608b0f9, State: Initialized, Role: LEARNER
I20240502 01:53:14.262513 4392 consensus_queue.cc:260] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 111, Last appended: 1.112, Last appended by leader: 112, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 105 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:14.269805 4392 ts_tablet_manager.cc:1398] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9: Time spent starting tablet: real 0.011s user 0.005s sys 0.004s
W20240502 01:53:14.324533 2187 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.198:37585, user_credentials={real_user=slave}} blocked reactor thread for 51052.3us
I20240502 01:53:14.336650 2275 tablet_copy_service.cc:342] P 446c7f9bf31a4acc9dae8daf43330906: Request end of tablet copy session 163649b5fbb140f18dccdf75f608b0f9-d592edf3298942cd93739aa846f867a4 received from {username='slave'} at 127.31.161.199:59365
I20240502 01:53:14.336823 2275 tablet_copy_service.cc:433] P 446c7f9bf31a4acc9dae8daf43330906: ending tablet copy session 163649b5fbb140f18dccdf75f608b0f9-d592edf3298942cd93739aa846f867a4 on tablet d592edf3298942cd93739aa846f867a4 with peer 163649b5fbb140f18dccdf75f608b0f9
W20240502 01:53:14.479408 2322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 51584.9us
I20240502 01:53:14.487715 3603 raft_consensus.cc:1203] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.104->[1.105-1.119] Dedup: 1.112->[1.113-1.119]
W20240502 01:53:14.497154 2583 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.197:38987, user_credentials={real_user=slave}} blocked reactor thread for 67716.1us
W20240502 01:53:14.569885 2322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 47743.8us
I20240502 01:53:14.754899 4401 raft_consensus.cc:1050] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906: attempting to promote NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 to VOTER
I20240502 01:53:14.755882 4401 consensus_queue.cc:237] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 1.123, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } }
I20240502 01:53:14.764186 1990 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 1 index: 124. (index mismatch)
I20240502 01:53:14.764300 3605 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 123. Preceding OpId from leader: term: 1 index: 124. (index mismatch)
I20240502 01:53:14.764863 4236 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20240502 01:53:14.766270 4236 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 122, Time since last communication: 0.000s
I20240502 01:53:14.779080 2387 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 123. Preceding OpId from leader: term: 1 index: 124. (index mismatch)
I20240502 01:53:14.787320 2514 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 123. Preceding OpId from leader: term: 1 index: 124. (index mismatch)
I20240502 01:53:14.801266 4236 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20240502 01:53:14.805373 4236 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20240502 01:53:14.818181 4401 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEADER]: Committing config change with OpId 1.124: config changed from index 105 to 124, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:14.821303 2516 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.124: config changed from index 105 to 124, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:14.827646 1841 catalog_manager.cc:5484] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 reported cstate change: config changed from index 105 to 124, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "446c7f9bf31a4acc9dae8daf43330906" committed_config { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:14.828207 1988 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.124: config changed from index 105 to 124, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:14.844367 3603 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.124: config changed from index 105 to 124, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:14.855073 2648 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 121. Preceding OpId from leader: term: 1 index: 124. (index mismatch)
I20240502 01:53:14.859722 4401 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 121, Time since last communication: 0.000s
I20240502 01:53:14.866830 2255 consensus_queue.cc:237] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 124, Committed index: 124, Last appended: 1.124, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } }
I20240502 01:53:14.874512 2385 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 125. (index mismatch)
I20240502 01:53:14.877885 1988 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 125. (index mismatch)
I20240502 01:53:14.878468 4236 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20240502 01:53:14.879238 4236 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 123, Time since last communication: 0.000s
I20240502 01:53:14.879362 2648 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.124: config changed from index 105 to 124, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:14.882980 2385 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.124: config changed from index 105 to 124, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:14.894930 4170 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEADER]: Committing config change with OpId 1.125: config changed from index 124 to 125, VOTER 9a3448a425c041258991dc60dd64ae7d (127.31.161.198) evicted. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:14.900241 2383 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 124 to 125, VOTER 9a3448a425c041258991dc60dd64ae7d (127.31.161.198) evicted. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:14.903299 1834 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d592edf3298942cd93739aa846f867a4 with cas_config_opid_index 124: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:14.902925 1987 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 124 to 125, VOTER 9a3448a425c041258991dc60dd64ae7d (127.31.161.198) evicted. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:14.906420 1848 catalog_manager.cc:5484] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 reported cstate change: config changed from index 124 to 125, VOTER 9a3448a425c041258991dc60dd64ae7d (127.31.161.198) evicted. New cstate: current_term: 1 leader_uuid: "446c7f9bf31a4acc9dae8daf43330906" committed_config { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:14.919960 2516 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 125. (index mismatch)
I20240502 01:53:14.920078 2630 tablet_service.cc:1514] Processing DeleteTablet for tablet d592edf3298942cd93739aa846f867a4 with delete_type TABLET_DATA_TOMBSTONED (TS 9a3448a425c041258991dc60dd64ae7d not found in new config with opid_index 125) from {username='slave'} at 127.0.0.1:52494
I20240502 01:53:14.920755 4454 tablet_replica.cc:330] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d: stopping tablet replica
I20240502 01:53:14.921052 4454 raft_consensus.cc:2229] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:14.921532 4170 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20240502 01:53:14.922783 4454 raft_consensus.cc:2258] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:14.924105 2516 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 124 to 125, VOTER 9a3448a425c041258991dc60dd64ae7d (127.31.161.198) evicted. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:14.925516 4454 ts_tablet_manager.cc:1875] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:14.938838 4454 ts_tablet_manager.cc:1888] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.124
I20240502 01:53:14.940125 4454 log.cc:1196] T d592edf3298942cd93739aa846f867a4 P 9a3448a425c041258991dc60dd64ae7d: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal/wals/d592edf3298942cd93739aa846f867a4
I20240502 01:53:14.945336 3605 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 124. Preceding OpId from leader: term: 1 index: 125. (index mismatch)
I20240502 01:53:14.946144 4170 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 125, Last known committed idx: 124, Time since last communication: 0.000s
I20240502 01:53:14.946774 1834 catalog_manager.cc:4830] TS 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585): tablet d592edf3298942cd93739aa846f867a4 (table rebalance_test_table_1 [id=db056d9436d84c2e8d32bb9d34b3c3a1]) successfully deleted
I20240502 01:53:14.948107 3602 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.125: config changed from index 124 to 125, VOTER 9a3448a425c041258991dc60dd64ae7d (127.31.161.198) evicted. New config: { opid_index: 125 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
W20240502 01:53:15.274673 2454 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 83337.8us
W20240502 01:53:15.284590 2319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.195:39353, user_credentials={real_user=slave}} blocked reactor thread for 147468us
W20240502 01:53:15.464358 2453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 50744.8us
W20240502 01:53:15.499368 2454 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 83098.2us
W20240502 01:53:15.797047 2453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 44587.2us
W20240502 01:53:15.942807 2451 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.195:39353, user_credentials={real_user=slave}} blocked reactor thread for 56393us
W20240502 01:53:15.946192 2453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 68994.1us
W20240502 01:53:16.242657 2454 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 46889.7us
I20240502 01:53:16.739403 2121 consensus_queue.cc:237] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:16.778014 2644 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:16.778091 2516 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:16.778664 3972 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:16.779264 3972 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:16.790117 2383 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:16.790640 3972 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:16.815485 3972 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:16.821760 2646 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:16.822746 1844 catalog_manager.cc:5484] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea reported cstate change: config changed from index -1 to 2, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New cstate: current_term: 1 leader_uuid: "269c9061e1a1435e92f79f0b9486dfea" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:16.831243 2516 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:16.917068 1988 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:53:16.918134 3972 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 01:53:16.930795 1988 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
W20240502 01:53:16.933018 2058 consensus_peers.cc:485] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea -> Peer 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381): Couldn't send request to peer 163649b5fbb140f18dccdf75f608b0f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74753a745d874fcea0fa9dfc4649aa14. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:16.935856 2383 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:17.078126 2646 consensus_queue.cc:237] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 137, Committed index: 137, Last appended: 1.138, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:17.119971 2124 raft_consensus.cc:1261] T 9895c93b27764c22b77087d74e0149c6 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 138. Preceding OpId from leader: term: 1 index: 139. (index mismatch)
I20240502 01:53:17.121260 4507 consensus_queue.cc:1035] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 139, Last known committed idx: 137, Time since last communication: 0.000s
W20240502 01:53:17.123900 2585 consensus_peers.cc:485] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d -> Peer 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381): Couldn't send request to peer 163649b5fbb140f18dccdf75f608b0f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9895c93b27764c22b77087d74e0149c6. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:17.124987 2516 raft_consensus.cc:1261] T 9895c93b27764c22b77087d74e0149c6 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 138. Preceding OpId from leader: term: 1 index: 139. (index mismatch)
I20240502 01:53:17.125695 4473 consensus_queue.cc:1035] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 139, Last known committed idx: 137, Time since last communication: 0.000s
I20240502 01:53:17.129838 2253 raft_consensus.cc:1261] T 9895c93b27764c22b77087d74e0149c6 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 138. Preceding OpId from leader: term: 1 index: 139. (index mismatch)
I20240502 01:53:17.130941 4405 consensus_queue.cc:1035] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 139, Last known committed idx: 137, Time since last communication: 0.000s
I20240502 01:53:17.130810 2514 raft_consensus.cc:2932] T 9895c93b27764c22b77087d74e0149c6 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.139: config changed from index -1 to 139, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:17.133761 4473 raft_consensus.cc:2932] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [term 1 LEADER]: Committing config change with OpId 1.139: config changed from index -1 to 139, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:17.134065 2253 raft_consensus.cc:2932] T 9895c93b27764c22b77087d74e0149c6 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.139: config changed from index -1 to 139, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:17.142725 1844 catalog_manager.cc:5484] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d reported cstate change: config changed from index -1 to 139, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New cstate: current_term: 1 leader_uuid: "9a3448a425c041258991dc60dd64ae7d" committed_config { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:17.150341 2384 raft_consensus.cc:1261] T 9895c93b27764c22b77087d74e0149c6 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 138. Preceding OpId from leader: term: 1 index: 139. (index mismatch)
I20240502 01:53:17.153381 2124 raft_consensus.cc:2932] T 9895c93b27764c22b77087d74e0149c6 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.139: config changed from index -1 to 139, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
W20240502 01:53:17.198189 2454 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 50979.2us
I20240502 01:53:17.226869 4435 consensus_queue.cc:1035] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 139, Last known committed idx: 137, Time since last communication: 0.000s
I20240502 01:53:17.285626 4539 ts_tablet_manager.cc:902] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9: Initiating tablet copy from peer 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621)
I20240502 01:53:17.318468 4539 tablet_copy_client.cc:308] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.194:39621
I20240502 01:53:17.388136 2385 raft_consensus.cc:2932] T 9895c93b27764c22b77087d74e0149c6 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.139: config changed from index -1 to 139, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:17.389647 2144 tablet_copy_service.cc:140] P 269c9061e1a1435e92f79f0b9486dfea: Received BeginTabletCopySession request for tablet 74753a745d874fcea0fa9dfc4649aa14 from peer 163649b5fbb140f18dccdf75f608b0f9 ({username='slave'} at 127.31.161.199:56791)
I20240502 01:53:17.389956 2144 tablet_copy_service.cc:161] P 269c9061e1a1435e92f79f0b9486dfea: Beginning new tablet copy session on tablet 74753a745d874fcea0fa9dfc4649aa14 from peer 163649b5fbb140f18dccdf75f608b0f9 at {username='slave'} at 127.31.161.199:56791: session id = 163649b5fbb140f18dccdf75f608b0f9-74753a745d874fcea0fa9dfc4649aa14
I20240502 01:53:17.404873 2144 tablet_copy_source_session.cc:205] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:17.423467 4539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74753a745d874fcea0fa9dfc4649aa14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:17.429891 4539 tablet_copy_client.cc:783] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:17.430184 4539 tablet_copy_client.cc:647] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:17.437067 4539 tablet_copy_client.cc:523] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:17.440822 4539 tablet_bootstrap.cc:492] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap starting.
W20240502 01:53:17.525836 2321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.199:46381, user_credentials={real_user=slave}} blocked reactor thread for 151278us
I20240502 01:53:17.712539 2384 consensus_queue.cc:237] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 1.110, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:17.729343 2249 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 1 index: 111. (index mismatch)
I20240502 01:53:17.729496 2120 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 1 index: 111. (index mismatch)
I20240502 01:53:17.730015 4265 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20240502 01:53:17.730463 4437 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20240502 01:53:17.733134 4500 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Committing config change with OpId 1.111: config changed from index -1 to 111, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:17.736388 1988 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 1 index: 111. (index mismatch)
I20240502 01:53:17.737985 2120 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.111: config changed from index -1 to 111, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:17.739745 4539 tablet_bootstrap.cc:492] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:17.740226 4539 tablet_bootstrap.cc:492] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap complete.
I20240502 01:53:17.740507 4539 ts_tablet_manager.cc:1367] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9: Time spent bootstrapping tablet: real 0.300s user 0.033s sys 0.066s
I20240502 01:53:17.740630 2249 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.111: config changed from index -1 to 111, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:17.741343 4539 raft_consensus.cc:348] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:17.741648 4539 raft_consensus.cc:729] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 163649b5fbb140f18dccdf75f608b0f9, State: Initialized, Role: LEARNER
I20240502 01:53:17.741878 4539 consensus_queue.cc:260] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:17.749647 1844 catalog_manager.cc:5484] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906 reported cstate change: config changed from index -1 to 111, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New cstate: current_term: 1 leader_uuid: "7ef43ea75ecb46ee905b781c91aed95a" committed_config { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:17.754101 4539 ts_tablet_manager.cc:1398] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9: Time spent starting tablet: real 0.013s user 0.006s sys 0.003s
I20240502 01:53:17.755156 2144 tablet_copy_service.cc:342] P 269c9061e1a1435e92f79f0b9486dfea: Request end of tablet copy session 163649b5fbb140f18dccdf75f608b0f9-74753a745d874fcea0fa9dfc4649aa14 received from {username='slave'} at 127.31.161.199:56791
I20240502 01:53:17.755373 2144 tablet_copy_service.cc:433] P 269c9061e1a1435e92f79f0b9486dfea: ending tablet copy session 163649b5fbb140f18dccdf75f608b0f9-74753a745d874fcea0fa9dfc4649aa14 on tablet 74753a745d874fcea0fa9dfc4649aa14 with peer 163649b5fbb140f18dccdf75f608b0f9
I20240502 01:53:17.759117 4539 ts_tablet_manager.cc:902] T 9895c93b27764c22b77087d74e0149c6 P 163649b5fbb140f18dccdf75f608b0f9: Initiating tablet copy from peer 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585)
W20240502 01:53:17.780529 2451 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.198:37585, user_credentials={real_user=slave}} blocked reactor thread for 79733.8us
W20240502 01:53:17.781031 2454 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 76468.1us
W20240502 01:53:17.793525 2453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 93064.3us
I20240502 01:53:17.795166 4539 tablet_copy_client.cc:308] T 9895c93b27764c22b77087d74e0149c6 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.198:37585
I20240502 01:53:17.796211 2670 tablet_copy_service.cc:140] P 9a3448a425c041258991dc60dd64ae7d: Received BeginTabletCopySession request for tablet 9895c93b27764c22b77087d74e0149c6 from peer 163649b5fbb140f18dccdf75f608b0f9 ({username='slave'} at 127.31.161.199:56747)
I20240502 01:53:17.796363 2670 tablet_copy_service.cc:161] P 9a3448a425c041258991dc60dd64ae7d: Beginning new tablet copy session on tablet 9895c93b27764c22b77087d74e0149c6 from peer 163649b5fbb140f18dccdf75f608b0f9 at {username='slave'} at 127.31.161.199:56747: session id = 163649b5fbb140f18dccdf75f608b0f9-9895c93b27764c22b77087d74e0149c6
I20240502 01:53:17.799167 2670 tablet_copy_source_session.cc:205] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:17.800740 4539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9895c93b27764c22b77087d74e0149c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:17.824482 4539 tablet_copy_client.cc:783] T 9895c93b27764c22b77087d74e0149c6 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:17.825822 4539 tablet_copy_client.cc:647] T 9895c93b27764c22b77087d74e0149c6 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:17.831861 4539 tablet_copy_client.cc:523] T 9895c93b27764c22b77087d74e0149c6 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:17.835392 4539 tablet_bootstrap.cc:492] T 9895c93b27764c22b77087d74e0149c6 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap starting.
I20240502 01:53:17.836750 2514 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 1 index: 111. (index mismatch)
W20240502 01:53:17.902141 2321 consensus_peers.cc:485] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a -> Peer 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381): Couldn't send request to peer 163649b5fbb140f18dccdf75f608b0f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0225b684df4f41499f76fe10b72a8861. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:17.903928 4265 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20240502 01:53:17.907574 1988 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.111: config changed from index -1 to 111, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:17.908015 4265 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20240502 01:53:17.909716 2514 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.111: config changed from index -1 to 111, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
W20240502 01:53:18.051477 2451 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.198:37585, user_credentials={real_user=slave}} blocked reactor thread for 96214.2us
I20240502 01:53:18.178859 2517 consensus_queue.cc:237] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.106, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:18.189337 4557 ts_tablet_manager.cc:902] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9: Initiating tablet copy from peer 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225)
I20240502 01:53:18.245808 2121 raft_consensus.cc:1261] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240502 01:53:18.246748 4587 consensus_queue.cc:1035] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
W20240502 01:53:18.252257 2451 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.195:39353, user_credentials={real_user=slave}} blocked reactor thread for 197923us
I20240502 01:53:18.284759 2649 raft_consensus.cc:1261] T 4068710224254825bf42671418592ece P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240502 01:53:18.287722 3688 consensus_queue.cc:1035] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 105, Time since last communication: 0.000s
I20240502 01:53:18.289292 2253 raft_consensus.cc:1261] T 4068710224254825bf42671418592ece P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240502 01:53:18.306002 3688 consensus_queue.cc:1035] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 105, Time since last communication: 0.000s
I20240502 01:53:18.307673 4557 tablet_copy_client.cc:308] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:39225
I20240502 01:53:18.310631 2408 tablet_copy_service.cc:140] P 7ef43ea75ecb46ee905b781c91aed95a: Received BeginTabletCopySession request for tablet 0225b684df4f41499f76fe10b72a8861 from peer 163649b5fbb140f18dccdf75f608b0f9 ({username='slave'} at 127.31.161.199:46507)
I20240502 01:53:18.310784 2408 tablet_copy_service.cc:161] P 7ef43ea75ecb46ee905b781c91aed95a: Beginning new tablet copy session on tablet 0225b684df4f41499f76fe10b72a8861 from peer 163649b5fbb140f18dccdf75f608b0f9 at {username='slave'} at 127.31.161.199:46507: session id = 163649b5fbb140f18dccdf75f608b0f9-0225b684df4f41499f76fe10b72a8861
I20240502 01:53:18.314136 2408 tablet_copy_source_session.cc:205] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:18.324110 4557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0225b684df4f41499f76fe10b72a8861. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:18.326330 1991 raft_consensus.cc:1261] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 106. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240502 01:53:18.329645 4577 raft_consensus.cc:2932] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [term 1 LEADER]: Committing config change with OpId 1.107: config changed from index -1 to 107, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:18.334657 2644 raft_consensus.cc:2932] T 4068710224254825bf42671418592ece P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index -1 to 107, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:18.343659 1841 catalog_manager.cc:5484] T 4068710224254825bf42671418592ece P 9a3448a425c041258991dc60dd64ae7d reported cstate change: config changed from index -1 to 107, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New cstate: current_term: 1 leader_uuid: "4d99e1fdda704211b3d101f892c57119" committed_config { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:18.345479 2253 raft_consensus.cc:2932] T 4068710224254825bf42671418592ece P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index -1 to 107, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:18.356526 4573 consensus_queue.cc:1035] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 107, Last known committed idx: 106, Time since last communication: 0.000s
I20240502 01:53:18.401096 4557 tablet_copy_client.cc:783] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:18.401367 4557 tablet_copy_client.cc:647] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:18.407678 4557 tablet_copy_client.cc:523] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:18.411031 4557 tablet_bootstrap.cc:492] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap starting.
W20240502 01:53:18.411947 2453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 55493.5us
I20240502 01:53:18.419252 2124 raft_consensus.cc:2932] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index -1 to 107, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:18.423270 1990 raft_consensus.cc:2932] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.107: config changed from index -1 to 107, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:18.434640 3603 raft_consensus.cc:1203] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240502 01:53:18.555604 2453 consensus_peers.cc:485] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 -> Peer 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381): Couldn't send request to peer 163649b5fbb140f18dccdf75f608b0f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4068710224254825bf42671418592ece. This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:18.676923 2451 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.198:37585, user_credentials={real_user=slave}} blocked reactor thread for 119473us
W20240502 01:53:18.677875 2453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 98934.5us
I20240502 01:53:18.805585 4607 raft_consensus.cc:1050] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea: attempting to promote NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 to VOTER
I20240502 01:53:18.806830 4607 consensus_queue.cc:237] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } }
I20240502 01:53:18.819166 2646 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:18.819979 3601 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:18.820165 1987 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:18.820924 2514 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:18.821012 4593 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:18.821585 4593 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:18.822106 4593 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:18.822548 4593 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 01:53:18.843252 2384 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 01:53:18.843859 4607 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 01:53:18.863624 2454 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 318143us
I20240502 01:53:18.883131 4557 tablet_bootstrap.cc:492] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=111 overwritten=0 applied=111 ignored=0} inserts{seen=796 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:18.883767 4557 tablet_bootstrap.cc:492] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap complete.
I20240502 01:53:18.884174 4557 ts_tablet_manager.cc:1367] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9: Time spent bootstrapping tablet: real 0.473s user 0.161s sys 0.022s
I20240502 01:53:18.885062 4557 raft_consensus.cc:348] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:18.885766 4557 raft_consensus.cc:729] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 163649b5fbb140f18dccdf75f608b0f9, State: Initialized, Role: LEARNER
I20240502 01:53:18.886045 4557 consensus_queue.cc:260] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 111, Last appended: 1.111, Last appended by leader: 111, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:18.889797 4557 ts_tablet_manager.cc:1398] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:53:18.892743 2408 tablet_copy_service.cc:342] P 7ef43ea75ecb46ee905b781c91aed95a: Request end of tablet copy session 163649b5fbb140f18dccdf75f608b0f9-0225b684df4f41499f76fe10b72a8861 received from {username='slave'} at 127.31.161.199:46507
I20240502 01:53:18.892900 2408 tablet_copy_service.cc:433] P 7ef43ea75ecb46ee905b781c91aed95a: ending tablet copy session 163649b5fbb140f18dccdf75f608b0f9-0225b684df4f41499f76fe10b72a8861 on tablet 0225b684df4f41499f76fe10b72a8861 with peer 163649b5fbb140f18dccdf75f608b0f9
I20240502 01:53:18.948660 4607 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:18.957406 1841 catalog_manager.cc:5484] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea reported cstate change: config changed from index 2 to 3, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "269c9061e1a1435e92f79f0b9486dfea" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:18.962702 2514 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:18.964766 2648 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:18.961473 3603 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:19.006531 1990 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:19.006111 2384 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
W20240502 01:53:19.042903 2189 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.199:46381, user_credentials={real_user=slave}} blocked reactor thread for 99343.5us
I20240502 01:53:19.044189 4557 ts_tablet_manager.cc:902] T 4068710224254825bf42671418592ece P 163649b5fbb140f18dccdf75f608b0f9: Initiating tablet copy from peer 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987)
I20240502 01:53:19.117558 2119 consensus_queue.cc:237] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } }
I20240502 01:53:19.196465 3603 raft_consensus.cc:1203] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.110->[1.111-1.117] Dedup: 1.111->[1.112-1.117]
W20240502 01:53:19.222369 2057 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.196:39225, user_credentials={real_user=slave}} blocked reactor thread for 57489.6us
W20240502 01:53:19.253417 3803 log.cc:924] Time spent T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20240502 01:53:19.283130 3600 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:19.284736 4555 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:19.286897 2514 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:19.288077 4555 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:19.290261 4607 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:19.290779 2516 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:19.296403 1847 catalog_manager.cc:5484] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119 reported cstate change: config changed from index 3 to 4, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "269c9061e1a1435e92f79f0b9486dfea" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:19.301004 1971 tablet_service.cc:1514] Processing DeleteTablet for tablet 74753a745d874fcea0fa9dfc4649aa14 with delete_type TABLET_DATA_TOMBSTONED (TS 58dae5dc94444d5e92e6907c5b637879 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42016
I20240502 01:53:19.304695 1837 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 74753a745d874fcea0fa9dfc4649aa14 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:19.305317 3603 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:19.314365 2648 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:19.315048 4555 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:19.316692 2646 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
W20240502 01:53:19.319272 2319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.198:37585, user_credentials={real_user=slave}} blocked reactor thread for 77489.4us
W20240502 01:53:19.319900 2321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.199:46381, user_credentials={real_user=slave}} blocked reactor thread for 63670.3us
I20240502 01:53:19.320441 2383 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:53:19.324004 4555 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:53:19.328922 2383 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:19.335317 4500 raft_consensus.cc:1050] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a: attempting to promote NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 to VOTER
I20240502 01:53:19.336006 4500 consensus_queue.cc:237] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 1.118, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } }
I20240502 01:53:19.352995 1991 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 118. Preceding OpId from leader: term: 1 index: 119. (index mismatch)
I20240502 01:53:19.353106 3601 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 118. Preceding OpId from leader: term: 1 index: 119. (index mismatch)
I20240502 01:53:19.353729 4499 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20240502 01:53:19.354370 4265 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20240502 01:53:19.357723 2252 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 118. Preceding OpId from leader: term: 1 index: 119. (index mismatch)
I20240502 01:53:19.358496 4437 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 117, Time since last communication: 0.000s
I20240502 01:53:19.367658 2514 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 118. Preceding OpId from leader: term: 1 index: 119. (index mismatch)
I20240502 01:53:19.384356 4265 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Committing config change with OpId 1.119: config changed from index 111 to 119, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:19.389851 3601 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.119: config changed from index 111 to 119, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:19.391764 4379 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20240502 01:53:19.392009 2249 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.119: config changed from index 111 to 119, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:19.399084 1847 catalog_manager.cc:5484] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 reported cstate change: config changed from index 111 to 119, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ef43ea75ecb46ee905b781c91aed95a" committed_config { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:19.404065 2516 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.119: config changed from index 111 to 119, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:19.408556 4651 tablet_replica.cc:330] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879: stopping tablet replica
I20240502 01:53:19.409619 1991 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.119: config changed from index 111 to 119, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:19.446281 4651 raft_consensus.cc:2229] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:19.446676 4651 raft_consensus.cc:2258] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:19.454571 4651 ts_tablet_manager.cc:1875] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:19.460184 2123 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 118. Preceding OpId from leader: term: 1 index: 119. (index mismatch)
I20240502 01:53:19.460883 4437 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 117, Time since last communication: 0.000s
I20240502 01:53:19.467046 2123 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.119: config changed from index 111 to 119, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:19.488792 4557 tablet_copy_client.cc:308] T 4068710224254825bf42671418592ece P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.197:38987
I20240502 01:53:19.507283 4651 ts_tablet_manager.cc:1888] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 01:53:19.509238 4651 log.cc:1196] T 74753a745d874fcea0fa9dfc4649aa14 P 58dae5dc94444d5e92e6907c5b637879: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/74753a745d874fcea0fa9dfc4649aa14
I20240502 01:53:19.513288 1836 catalog_manager.cc:4830] TS 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): tablet 74753a745d874fcea0fa9dfc4649aa14 (table TestTable [id=21d6c8edffeb445cac75c180ed630ada]) successfully deleted
I20240502 01:53:19.531565 2539 tablet_copy_service.cc:140] P 4d99e1fdda704211b3d101f892c57119: Received BeginTabletCopySession request for tablet 4068710224254825bf42671418592ece from peer 163649b5fbb140f18dccdf75f608b0f9 ({username='slave'} at 127.31.161.199:33743)
I20240502 01:53:19.531767 2539 tablet_copy_service.cc:161] P 4d99e1fdda704211b3d101f892c57119: Beginning new tablet copy session on tablet 4068710224254825bf42671418592ece from peer 163649b5fbb140f18dccdf75f608b0f9 at {username='slave'} at 127.31.161.199:33743: session id = 163649b5fbb140f18dccdf75f608b0f9-4068710224254825bf42671418592ece
I20240502 01:53:19.534569 2539 tablet_copy_source_session.cc:205] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:19.536077 4557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4068710224254825bf42671418592ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:19.543686 4557 tablet_copy_client.cc:783] T 4068710224254825bf42671418592ece P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:19.544023 4557 tablet_copy_client.cc:647] T 4068710224254825bf42671418592ece P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:19.602396 4557 tablet_copy_client.cc:523] T 4068710224254825bf42671418592ece P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:19.607913 4557 tablet_bootstrap.cc:492] T 4068710224254825bf42671418592ece P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap starting.
I20240502 01:53:19.654361 2383 consensus_queue.cc:237] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 121, Committed index: 121, Last appended: 1.121, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } }
I20240502 01:53:19.660138 2121 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 121. Preceding OpId from leader: term: 1 index: 122. (index mismatch)
I20240502 01:53:19.660907 4499 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 120, Time since last communication: 0.000s
I20240502 01:53:19.666235 2514 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 121. Preceding OpId from leader: term: 1 index: 122. (index mismatch)
I20240502 01:53:19.670892 4379 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 121, Time since last communication: 0.000s
I20240502 01:53:19.672607 1988 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 120. Preceding OpId from leader: term: 1 index: 122. (index mismatch)
I20240502 01:53:19.675746 3604 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 121. Preceding OpId from leader: term: 1 index: 122. (index mismatch)
I20240502 01:53:19.677306 4499 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Committing config change with OpId 1.122: config changed from index 119 to 122, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:19.678563 2514 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.122: config changed from index 119 to 122, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:19.692086 1835 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0225b684df4f41499f76fe10b72a8861 with cas_config_opid_index 119: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:19.692875 2121 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.122: config changed from index 119 to 122, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:19.694161 4437 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 120, Time since last communication: 0.000s
I20240502 01:53:19.696997 4500 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 122, Last known committed idx: 120, Time since last communication: 0.000s
I20240502 01:53:19.700151 1988 raft_consensus.cc:1203] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.120->[1.121-1.122] Dedup: 1.121->[1.122-1.122]
I20240502 01:53:19.701309 3605 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.122: config changed from index 119 to 122, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:19.703907 1988 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.122: config changed from index 119 to 122, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New config: { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:19.705050 1845 catalog_manager.cc:5484] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 reported cstate change: config changed from index 119 to 122, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New cstate: current_term: 1 leader_uuid: "7ef43ea75ecb46ee905b781c91aed95a" committed_config { opid_index: 122 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:19.712812 2234 tablet_service.cc:1514] Processing DeleteTablet for tablet 0225b684df4f41499f76fe10b72a8861 with delete_type TABLET_DATA_TOMBSTONED (TS 446c7f9bf31a4acc9dae8daf43330906 not found in new config with opid_index 122) from {username='slave'} at 127.0.0.1:45438
I20240502 01:53:19.756760 4663 tablet_replica.cc:330] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906: stopping tablet replica
I20240502 01:53:19.758795 4663 raft_consensus.cc:2229] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:19.763437 4663 raft_consensus.cc:2258] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:19.767809 4663 ts_tablet_manager.cc:1875] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:19.863278 4663 ts_tablet_manager.cc:1888] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.121
I20240502 01:53:19.876794 4663 log.cc:1196] T 0225b684df4f41499f76fe10b72a8861 P 446c7f9bf31a4acc9dae8daf43330906: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/wals/0225b684df4f41499f76fe10b72a8861
I20240502 01:53:19.885879 1834 catalog_manager.cc:4830] TS 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353): tablet 0225b684df4f41499f76fe10b72a8861 (table rebalance_test_table_2 [id=832ef7a5c81541c3a3c9b450b6af8fc1]) successfully deleted
W20240502 01:53:20.010579 2585 consensus_peers.cc:485] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d -> Peer 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381): Couldn't send request to peer 163649b5fbb140f18dccdf75f608b0f9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240502 01:53:20.140003 2321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 118018us
W20240502 01:53:20.143306 2322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 68961.1us
W20240502 01:53:20.160009 2319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.195:39353, user_credentials={real_user=slave}} blocked reactor thread for 87758.4us
W20240502 01:53:20.228148 2321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 44923.3us
W20240502 01:53:20.253980 2319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.198:37585, user_credentials={real_user=slave}} blocked reactor thread for 65832.4us
W20240502 01:53:20.354751 2319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.197:38987, user_credentials={real_user=slave}} blocked reactor thread for 51566.6us
W20240502 01:53:20.536768 2322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 65368.1us
W20240502 01:53:20.750044 2322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 65856us
I20240502 01:53:20.799301 1990 consensus_queue.cc:237] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 194, Committed index: 194, Last appended: 1.194, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:20.804553 2115 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 194. Preceding OpId from leader: term: 1 index: 195. (index mismatch)
I20240502 01:53:20.826138 4596 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 195, Last known committed idx: 194, Time since last communication: 0.021s
I20240502 01:53:20.831202 2251 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 194. Preceding OpId from leader: term: 1 index: 195. (index mismatch)
I20240502 01:53:20.831754 2649 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 194. Preceding OpId from leader: term: 1 index: 195. (index mismatch)
I20240502 01:53:20.832408 4589 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 195, Last known committed idx: 194, Time since last communication: 0.000s
I20240502 01:53:20.836042 4589 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 195, Last known committed idx: 194, Time since last communication: 0.000s
I20240502 01:53:20.839154 4589 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [term 1 LEADER]: Committing config change with OpId 1.195: config changed from index -1 to 195, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:20.844568 2249 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.195: config changed from index -1 to 195, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:20.861644 2121 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.195: config changed from index -1 to 195, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:20.880200 1847 catalog_manager.cc:5484] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906 reported cstate change: config changed from index -1 to 195, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New cstate: current_term: 1 leader_uuid: "58dae5dc94444d5e92e6907c5b637879" committed_config { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
W20240502 01:53:20.905680 4539 log.cc:924] Time spent T 9895c93b27764c22b77087d74e0149c6 P 163649b5fbb140f18dccdf75f608b0f9: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
W20240502 01:53:20.909626 2322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 62087.6us
W20240502 01:53:20.914652 2319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.197:38987, user_credentials={real_user=slave}} blocked reactor thread for 59618.4us
I20240502 01:53:20.915340 2387 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 194. Preceding OpId from leader: term: 1 index: 195. (index mismatch)
I20240502 01:53:20.930673 4290 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 195, Last known committed idx: 194, Time since last communication: 0.000s
I20240502 01:53:20.932475 4557 tablet_bootstrap.cc:492] T 4068710224254825bf42671418592ece P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=113 overwritten=0 applied=113 ignored=0} inserts{seen=785 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:20.932984 4557 tablet_bootstrap.cc:492] T 4068710224254825bf42671418592ece P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap complete.
I20240502 01:53:20.933378 4557 ts_tablet_manager.cc:1367] T 4068710224254825bf42671418592ece P 163649b5fbb140f18dccdf75f608b0f9: Time spent bootstrapping tablet: real 1.326s user 0.157s sys 0.014s
I20240502 01:53:20.934291 4557 raft_consensus.cc:348] T 4068710224254825bf42671418592ece P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:20.934675 4557 raft_consensus.cc:729] T 4068710224254825bf42671418592ece P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 163649b5fbb140f18dccdf75f608b0f9, State: Initialized, Role: LEARNER
I20240502 01:53:20.935156 4557 consensus_queue.cc:260] T 4068710224254825bf42671418592ece P 163649b5fbb140f18dccdf75f608b0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 113, Last appended: 1.113, Last appended by leader: 113, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 107 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:20.940318 2383 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.195: config changed from index -1 to 195, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
I20240502 01:53:20.942675 4557 ts_tablet_manager.cc:1398] T 4068710224254825bf42671418592ece P 163649b5fbb140f18dccdf75f608b0f9: Time spent starting tablet: real 0.009s user 0.001s sys 0.006s
W20240502 01:53:20.943235 1927 consensus_peers.cc:485] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 -> Peer 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381): Couldn't send request to peer 163649b5fbb140f18dccdf75f608b0f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4abedcf755774255867d46154f8131c7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:20.944561 2539 tablet_copy_service.cc:342] P 4d99e1fdda704211b3d101f892c57119: Request end of tablet copy session 163649b5fbb140f18dccdf75f608b0f9-4068710224254825bf42671418592ece received from {username='slave'} at 127.31.161.199:33743
I20240502 01:53:20.944691 2539 tablet_copy_service.cc:433] P 4d99e1fdda704211b3d101f892c57119: ending tablet copy session 163649b5fbb140f18dccdf75f608b0f9-4068710224254825bf42671418592ece on tablet 4068710224254825bf42671418592ece with peer 163649b5fbb140f18dccdf75f608b0f9
I20240502 01:53:20.958992 2646 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.195: config changed from index -1 to 195, NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) added. New config: { opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } } }
W20240502 01:53:20.965185 2322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 53351.8us
I20240502 01:53:20.991989 4539 tablet_bootstrap.cc:492] T 9895c93b27764c22b77087d74e0149c6 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=143 overwritten=0 applied=143 ignored=0} inserts{seen=774 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:20.992571 4539 tablet_bootstrap.cc:492] T 9895c93b27764c22b77087d74e0149c6 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap complete.
I20240502 01:53:20.992929 4539 ts_tablet_manager.cc:1367] T 9895c93b27764c22b77087d74e0149c6 P 163649b5fbb140f18dccdf75f608b0f9: Time spent bootstrapping tablet: real 3.158s user 0.260s sys 0.096s
I20240502 01:53:20.993901 4539 raft_consensus.cc:348] T 9895c93b27764c22b77087d74e0149c6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:20.994282 4539 raft_consensus.cc:729] T 9895c93b27764c22b77087d74e0149c6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 163649b5fbb140f18dccdf75f608b0f9, State: Initialized, Role: LEARNER
I20240502 01:53:20.994606 4539 consensus_queue.cc:260] T 9895c93b27764c22b77087d74e0149c6 P 163649b5fbb140f18dccdf75f608b0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 143, Last appended: 1.143, Last appended by leader: 143, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:20.995263 4539 ts_tablet_manager.cc:1398] T 9895c93b27764c22b77087d74e0149c6 P 163649b5fbb140f18dccdf75f608b0f9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:53:20.996917 2670 tablet_copy_service.cc:342] P 9a3448a425c041258991dc60dd64ae7d: Request end of tablet copy session 163649b5fbb140f18dccdf75f608b0f9-9895c93b27764c22b77087d74e0149c6 received from {username='slave'} at 127.31.161.199:56747
I20240502 01:53:20.997056 2670 tablet_copy_service.cc:433] P 9a3448a425c041258991dc60dd64ae7d: ending tablet copy session 163649b5fbb140f18dccdf75f608b0f9-9895c93b27764c22b77087d74e0149c6 on tablet 9895c93b27764c22b77087d74e0149c6 with peer 163649b5fbb140f18dccdf75f608b0f9
I20240502 01:53:21.303551 3603 raft_consensus.cc:1203] T 4068710224254825bf42671418592ece P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.106->[1.107-1.122] Dedup: 1.113->[1.114-1.122]
I20240502 01:53:21.366508 3602 raft_consensus.cc:1203] T 9895c93b27764c22b77087d74e0149c6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.138->[1.139-1.176] Dedup: 1.143->[1.144-1.176]
I20240502 01:53:21.395844 4539 ts_tablet_manager.cc:902] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9: Initiating tablet copy from peer 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307)
I20240502 01:53:21.551744 4476 raft_consensus.cc:1050] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119: attempting to promote NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 to VOTER
I20240502 01:53:21.553507 4476 consensus_queue.cc:237] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 123, Committed index: 123, Last appended: 1.123, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } }
I20240502 01:53:21.556665 2254 raft_consensus.cc:1261] T 4068710224254825bf42671418592ece P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 123. Preceding OpId from leader: term: 1 index: 124. (index mismatch)
I20240502 01:53:21.557296 4573 consensus_queue.cc:1035] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20240502 01:53:21.567226 2647 raft_consensus.cc:1261] T 4068710224254825bf42671418592ece P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 123. Preceding OpId from leader: term: 1 index: 124. (index mismatch)
I20240502 01:53:21.570163 4573 consensus_queue.cc:1035] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20240502 01:53:21.601464 3603 raft_consensus.cc:1261] T 4068710224254825bf42671418592ece P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 123. Preceding OpId from leader: term: 1 index: 124. (index mismatch)
I20240502 01:53:21.602092 1990 raft_consensus.cc:1261] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 123. Preceding OpId from leader: term: 1 index: 124. (index mismatch)
I20240502 01:53:21.603232 4573 consensus_queue.cc:1035] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
W20240502 01:53:21.609128 2454 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 60425.5us
I20240502 01:53:21.617252 4305 consensus_queue.cc:1035] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 123, Time since last communication: 0.000s
I20240502 01:53:21.637141 4305 raft_consensus.cc:2932] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [term 1 LEADER]: Committing config change with OpId 1.124: config changed from index 107 to 124, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:21.642123 2251 raft_consensus.cc:2932] T 4068710224254825bf42671418592ece P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.124: config changed from index 107 to 124, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:21.643513 1845 catalog_manager.cc:5484] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 reported cstate change: config changed from index 107 to 124, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d99e1fdda704211b3d101f892c57119" committed_config { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:21.645500 3604 raft_consensus.cc:2932] T 4068710224254825bf42671418592ece P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.124: config changed from index 107 to 124, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:21.664176 4539 tablet_copy_client.cc:308] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.193:39307
I20240502 01:53:21.685442 1990 raft_consensus.cc:2932] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.124: config changed from index 107 to 124, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
W20240502 01:53:21.697572 2584 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.196:39225, user_credentials={real_user=slave}} blocked reactor thread for 106905us
I20240502 01:53:21.698349 2118 raft_consensus.cc:1261] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 1 index: 124. (index mismatch)
I20240502 01:53:21.706200 2647 raft_consensus.cc:2932] T 4068710224254825bf42671418592ece P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.124: config changed from index 107 to 124, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:21.731858 4580 consensus_queue.cc:1035] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 124, Last known committed idx: 121, Time since last communication: 0.000s
I20240502 01:53:21.736199 2114 raft_consensus.cc:2932] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.124: config changed from index 107 to 124, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 124 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:21.772907 4578 raft_consensus.cc:1050] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d: attempting to promote NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 to VOTER
I20240502 01:53:21.773891 4578 consensus_queue.cc:237] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 177, Committed index: 177, Last appended: 1.177, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 178 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } }
I20240502 01:53:21.776561 2249 raft_consensus.cc:1261] T 9895c93b27764c22b77087d74e0149c6 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 176. Preceding OpId from leader: term: 1 index: 178. (index mismatch)
I20240502 01:53:21.777134 4435 consensus_queue.cc:1035] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 178, Last known committed idx: 176, Time since last communication: 0.000s
I20240502 01:53:21.777428 3602 raft_consensus.cc:1261] T 9895c93b27764c22b77087d74e0149c6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 177. Preceding OpId from leader: term: 1 index: 178. (index mismatch)
I20240502 01:53:21.777980 4435 consensus_queue.cc:1035] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 178, Last known committed idx: 177, Time since last communication: 0.000s
I20240502 01:53:21.792667 2516 consensus_queue.cc:237] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 125, Committed index: 125, Last appended: 1.126, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } }
I20240502 01:53:21.807384 2254 raft_consensus.cc:1261] T 4068710224254825bf42671418592ece P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 126. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240502 01:53:21.808013 4580 consensus_queue.cc:1035] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 125, Time since last communication: 0.000s
I20240502 01:53:21.819658 3601 raft_consensus.cc:1261] T 4068710224254825bf42671418592ece P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 125. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240502 01:53:21.825518 2387 raft_consensus.cc:1261] T 9895c93b27764c22b77087d74e0149c6 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 177. Preceding OpId from leader: term: 1 index: 178. (index mismatch)
I20240502 01:53:21.826293 2647 raft_consensus.cc:1261] T 4068710224254825bf42671418592ece P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 126. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240502 01:53:21.827428 4750 consensus_queue.cc:1035] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 125, Time since last communication: 0.000s
I20240502 01:53:21.829624 4750 consensus_queue.cc:1035] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 124, Time since last communication: 0.001s
I20240502 01:53:21.831737 4405 consensus_queue.cc:1035] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 178, Last known committed idx: 177, Time since last communication: 0.000s
I20240502 01:53:21.833478 2012 tablet_copy_service.cc:140] P 58dae5dc94444d5e92e6907c5b637879: Received BeginTabletCopySession request for tablet 4abedcf755774255867d46154f8131c7 from peer 163649b5fbb140f18dccdf75f608b0f9 ({username='slave'} at 127.31.161.199:57725)
I20240502 01:53:21.833704 2012 tablet_copy_service.cc:161] P 58dae5dc94444d5e92e6907c5b637879: Beginning new tablet copy session on tablet 4abedcf755774255867d46154f8131c7 from peer 163649b5fbb140f18dccdf75f608b0f9 at {username='slave'} at 127.31.161.199:57725: session id = 163649b5fbb140f18dccdf75f608b0f9-4abedcf755774255867d46154f8131c7
I20240502 01:53:21.838461 2012 tablet_copy_source_session.cc:205] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:21.839658 4435 raft_consensus.cc:2932] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [term 1 LEADER]: Committing config change with OpId 1.178: config changed from index 139 to 178, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 178 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:21.840312 4539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4abedcf755774255867d46154f8131c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:21.846011 2387 raft_consensus.cc:2932] T 9895c93b27764c22b77087d74e0149c6 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.178: config changed from index 139 to 178, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 178 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:21.867923 4476 raft_consensus.cc:2932] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [term 1 LEADER]: Committing config change with OpId 1.127: config changed from index 124 to 127, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:21.868511 2646 raft_consensus.cc:2932] T 4068710224254825bf42671418592ece P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.127: config changed from index 124 to 127, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:21.877995 2249 raft_consensus.cc:2932] T 4068710224254825bf42671418592ece P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.127: config changed from index 124 to 127, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:21.883816 1847 catalog_manager.cc:5484] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 reported cstate change: config changed from index 124 to 127, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New cstate: current_term: 1 leader_uuid: "4d99e1fdda704211b3d101f892c57119" committed_config { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:21.888224 2103 tablet_service.cc:1514] Processing DeleteTablet for tablet 4068710224254825bf42671418592ece with delete_type TABLET_DATA_TOMBSTONED (TS 269c9061e1a1435e92f79f0b9486dfea not found in new config with opid_index 127) from {username='slave'} at 127.0.0.1:50686
W20240502 01:53:21.891896 2453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 58006.1us
I20240502 01:53:21.895735 4539 tablet_copy_client.cc:783] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:21.902287 4539 tablet_copy_client.cc:647] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:21.904814 2251 raft_consensus.cc:2932] T 9895c93b27764c22b77087d74e0149c6 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.178: config changed from index 139 to 178, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 178 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:21.910602 2121 raft_consensus.cc:1261] T 9895c93b27764c22b77087d74e0149c6 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 177. Preceding OpId from leader: term: 1 index: 179. (index mismatch)
I20240502 01:53:21.911609 4621 consensus_queue.cc:1035] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 178, Last known committed idx: 177, Time since last communication: 0.000s
I20240502 01:53:21.912603 1847 catalog_manager.cc:5484] T 9895c93b27764c22b77087d74e0149c6 P 446c7f9bf31a4acc9dae8daf43330906 reported cstate change: config changed from index 139 to 178, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9a3448a425c041258991dc60dd64ae7d" committed_config { opid_index: 178 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:21.914253 2114 raft_consensus.cc:2932] T 9895c93b27764c22b77087d74e0149c6 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.178: config changed from index 139 to 178, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 178 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:21.933614 2517 raft_consensus.cc:1261] T 9895c93b27764c22b77087d74e0149c6 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 176. Preceding OpId from leader: term: 1 index: 179. (index mismatch)
I20240502 01:53:21.927354 3604 raft_consensus.cc:2932] T 9895c93b27764c22b77087d74e0149c6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.178: config changed from index 139 to 178, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 178 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:21.933900 1834 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4068710224254825bf42671418592ece with cas_config_opid_index 124: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:21.937667 4768 tablet_replica.cc:330] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea: stopping tablet replica
I20240502 01:53:21.937467 3600 raft_consensus.cc:2932] T 4068710224254825bf42671418592ece P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.127: config changed from index 124 to 127, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:21.942281 4767 consensus_queue.cc:1035] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 178, Last known committed idx: 176, Time since last communication: 0.006s
I20240502 01:53:21.949545 1988 raft_consensus.cc:1261] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 125. Preceding OpId from leader: term: 1 index: 127. (index mismatch)
I20240502 01:53:21.953722 2517 raft_consensus.cc:2932] T 9895c93b27764c22b77087d74e0149c6 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.178: config changed from index 139 to 178, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 178 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:21.963353 4750 consensus_queue.cc:1035] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 127, Last known committed idx: 124, Time since last communication: 0.000s
I20240502 01:53:21.971966 4539 tablet_copy_client.cc:523] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:21.977607 4768 raft_consensus.cc:2229] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:21.978108 4768 pending_rounds.cc:62] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea: Trying to abort 1 pending ops.
I20240502 01:53:21.978462 4768 pending_rounds.cc:68] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea: Aborting op as it isn't in flight: id { term: 1 index: 126 } timestamp: 7023062228094013440 op_type: WRITE_OP write_request { tablet_id: "4068710224254825bf42671418592ece" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\033dR\r\320\363\220\025\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""fbb46ec785aa1488" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023062227950833664 authz_token { token_data: "\010\247\346\313\261\006\"3\n\005slave\022*\n 832ef7a5c81541c3a3c9b450b6af8fc1\020\001\030\001 \001(\001" signature: "<redacted>""z\241\000\235\201\013\225\245\261\262[\014\216d\340\016O\306\207\327\036\371\267\267p\366\211\203\225\003q.\r\265\021%/\257R\241\221\333G\264\373\377\326\234\334*\276.\001r\347S\200\325\237o=\247&\253" signing_key_seq_num: 0 } } request_id { client_id: "dd3ccb9081334271b6f36516c0aef9b1" seq_no: 368 first_incomplete_seq_no: 368 attempt_no: 0 }
W20240502 01:53:21.981981 2321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.199:46381, user_credentials={real_user=slave}} blocked reactor thread for 54891.8us
W20240502 01:53:21.986152 2322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 58634.7us
I20240502 01:53:22.038430 1990 raft_consensus.cc:2932] T 4068710224254825bf42671418592ece P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.127: config changed from index 124 to 127, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New config: { opid_index: 127 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:22.050016 2649 consensus_queue.cc:237] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 179, Committed index: 179, Last appended: 1.179, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } }
I20240502 01:53:22.054486 2251 raft_consensus.cc:1261] T 9895c93b27764c22b77087d74e0149c6 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 179. Preceding OpId from leader: term: 1 index: 180. (index mismatch)
I20240502 01:53:22.055099 4405 consensus_queue.cc:1035] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 180, Last known committed idx: 179, Time since last communication: 0.000s
I20240502 01:53:22.082267 4539 tablet_bootstrap.cc:492] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap starting.
I20240502 01:53:22.057983 2120 raft_consensus.cc:1261] T 9895c93b27764c22b77087d74e0149c6 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 179. Preceding OpId from leader: term: 1 index: 180. (index mismatch)
I20240502 01:53:22.093739 4578 consensus_queue.cc:1035] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 180, Last known committed idx: 179, Time since last communication: 0.000s
I20240502 01:53:22.107321 4405 raft_consensus.cc:2932] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [term 1 LEADER]: Committing config change with OpId 1.180: config changed from index 178 to 180, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New config: { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:22.107832 2120 raft_consensus.cc:2932] T 9895c93b27764c22b77087d74e0149c6 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.180: config changed from index 178 to 180, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New config: { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:22.113243 1834 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9895c93b27764c22b77087d74e0149c6 with cas_config_opid_index 178: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:22.114131 2251 raft_consensus.cc:2932] T 9895c93b27764c22b77087d74e0149c6 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.180: config changed from index 178 to 180, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New config: { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:22.120043 1844 catalog_manager.cc:5484] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d reported cstate change: config changed from index 178 to 180, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "9a3448a425c041258991dc60dd64ae7d" committed_config { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:22.129915 2516 raft_consensus.cc:1261] T 9895c93b27764c22b77087d74e0149c6 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 179. Preceding OpId from leader: term: 1 index: 180. (index mismatch)
I20240502 01:53:22.134457 2366 tablet_service.cc:1514] Processing DeleteTablet for tablet 9895c93b27764c22b77087d74e0149c6 with delete_type TABLET_DATA_TOMBSTONED (TS 7ef43ea75ecb46ee905b781c91aed95a not found in new config with opid_index 180) from {username='slave'} at 127.0.0.1:54148
I20240502 01:53:22.145637 4773 tablet_replica.cc:330] T 9895c93b27764c22b77087d74e0149c6 P 7ef43ea75ecb46ee905b781c91aed95a: stopping tablet replica
I20240502 01:53:22.176311 4768 raft_consensus.cc:2258] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 01:53:22.205690 2453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 84020.1us
I20240502 01:53:22.212131 4769 consensus_queue.cc:1035] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 180, Last known committed idx: 179, Time since last communication: 0.000s
I20240502 01:53:22.235345 4768 ts_tablet_manager.cc:1875] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:22.259171 2515 raft_consensus.cc:2932] T 9895c93b27764c22b77087d74e0149c6 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.180: config changed from index 178 to 180, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New config: { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:22.260494 4773 raft_consensus.cc:2229] T 9895c93b27764c22b77087d74e0149c6 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:22.262368 4773 raft_consensus.cc:2258] T 9895c93b27764c22b77087d74e0149c6 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:22.272770 4773 ts_tablet_manager.cc:1875] T 9895c93b27764c22b77087d74e0149c6 P 7ef43ea75ecb46ee905b781c91aed95a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:22.284518 3600 raft_consensus.cc:1261] T 9895c93b27764c22b77087d74e0149c6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 179. Preceding OpId from leader: term: 1 index: 180. (index mismatch)
I20240502 01:53:22.285431 4765 consensus_queue.cc:1035] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 180, Last known committed idx: 178, Time since last communication: 0.000s
I20240502 01:53:22.294354 3600 raft_consensus.cc:2932] T 9895c93b27764c22b77087d74e0149c6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.180: config changed from index 178 to 180, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New config: { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:22.352324 4773 ts_tablet_manager.cc:1888] T 9895c93b27764c22b77087d74e0149c6 P 7ef43ea75ecb46ee905b781c91aed95a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.179
I20240502 01:53:22.370534 4768 ts_tablet_manager.cc:1888] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.126
I20240502 01:53:22.370731 4768 log.cc:1196] T 4068710224254825bf42671418592ece P 269c9061e1a1435e92f79f0b9486dfea: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/wals/4068710224254825bf42671418592ece
I20240502 01:53:22.371590 1837 catalog_manager.cc:4830] TS 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621): tablet 4068710224254825bf42671418592ece (table rebalance_test_table_2 [id=832ef7a5c81541c3a3c9b450b6af8fc1]) successfully deleted
I20240502 01:53:22.380487 4773 log.cc:1196] T 9895c93b27764c22b77087d74e0149c6 P 7ef43ea75ecb46ee905b781c91aed95a: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/9895c93b27764c22b77087d74e0149c6
I20240502 01:53:22.381675 1835 catalog_manager.cc:4830] TS 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225): tablet 9895c93b27764c22b77087d74e0149c6 (table rebalance_test_table_1 [id=db056d9436d84c2e8d32bb9d34b3c3a1]) successfully deleted
W20240502 01:53:23.061600 4539 log.cc:924] Time spent T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9: Append to log took a long time: real 0.089s user 0.004s sys 0.000s
W20240502 01:53:23.403641 2586 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 78937us
W20240502 01:53:23.484390 4539 log.cc:924] Time spent T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9: Append to log took a long time: real 0.200s user 0.001s sys 0.000s
W20240502 01:53:23.882512 2453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 119134us
W20240502 01:53:24.097484 2454 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 111479us
I20240502 01:53:24.163753 4539 tablet_bootstrap.cc:492] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=206 overwritten=0 applied=206 ignored=0} inserts{seen=837 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:24.164366 4539 tablet_bootstrap.cc:492] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9: Bootstrap complete.
I20240502 01:53:24.164774 4539 ts_tablet_manager.cc:1367] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9: Time spent bootstrapping tablet: real 2.083s user 0.261s sys 0.027s
W20240502 01:53:24.198210 1927 consensus_peers.cc:485] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 -> Peer 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381): Couldn't send request to peer 163649b5fbb140f18dccdf75f608b0f9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240502 01:53:24.255522 4539 raft_consensus.cc:348] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
I20240502 01:53:24.256093 4539 raft_consensus.cc:729] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 163649b5fbb140f18dccdf75f608b0f9, State: Initialized, Role: LEARNER
I20240502 01:53:24.256412 4539 consensus_queue.cc:260] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 206, Last appended: 1.206, Last appended by leader: 206, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 195 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: true } }
W20240502 01:53:24.273169 2453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 55831.4us
I20240502 01:53:24.358497 4539 ts_tablet_manager.cc:1398] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9: Time spent starting tablet: real 0.194s user 0.000s sys 0.002s
I20240502 01:53:24.359660 2012 tablet_copy_service.cc:342] P 58dae5dc94444d5e92e6907c5b637879: Request end of tablet copy session 163649b5fbb140f18dccdf75f608b0f9-4abedcf755774255867d46154f8131c7 received from {username='slave'} at 127.31.161.199:57725
I20240502 01:53:24.359850 2012 tablet_copy_service.cc:433] P 58dae5dc94444d5e92e6907c5b637879: ending tablet copy session 163649b5fbb140f18dccdf75f608b0f9-4abedcf755774255867d46154f8131c7 on tablet 4abedcf755774255867d46154f8131c7 with peer 163649b5fbb140f18dccdf75f608b0f9
W20240502 01:53:24.456532 2319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.195:39353, user_credentials={real_user=slave}} blocked reactor thread for 58584.7us
I20240502 01:53:24.782315 3601 raft_consensus.cc:1203] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.194->[1.195-1.232] Dedup: 1.206->[1.207-1.232]
W20240502 01:53:24.889169 2453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 46593.5us
I20240502 01:53:25.106151 4857 raft_consensus.cc:1050] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879: attempting to promote NON_VOTER 163649b5fbb140f18dccdf75f608b0f9 to VOTER
I20240502 01:53:25.107483 4857 consensus_queue.cc:237] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 234, Committed index: 234, Last appended: 1.234, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } }
I20240502 01:53:25.118350 2120 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 1 index: 235. (index mismatch)
I20240502 01:53:25.118887 3604 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 LEARNER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 1 index: 235. (index mismatch)
I20240502 01:53:25.119139 4859 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20240502 01:53:25.119470 4596 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20240502 01:53:25.121883 2387 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 1 index: 235. (index mismatch)
I20240502 01:53:25.122886 4596 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20240502 01:53:25.125880 4596 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [term 1 LEADER]: Committing config change with OpId 1.235: config changed from index 195 to 235, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:25.129748 2383 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.235: config changed from index 195 to 235, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:25.135335 1848 catalog_manager.cc:5484] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a reported cstate change: config changed from index 195 to 235, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58dae5dc94444d5e92e6907c5b637879" committed_config { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:25.137368 3604 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.235: config changed from index 195 to 235, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:25.137672 2114 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.235: config changed from index 195 to 235, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:25.146288 2251 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 1 index: 235. (index mismatch)
I20240502 01:53:25.147011 4857 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20240502 01:53:25.149561 2645 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 1 index: 235. (index mismatch)
I20240502 01:53:25.150135 4857 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20240502 01:53:25.151618 2646 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.235: config changed from index 195 to 235, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:25.151801 2251 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.235: config changed from index 195 to 235, 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:25.504016 1989 consensus_queue.cc:237] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 238, Committed index: 238, Last appended: 1.238, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } }
I20240502 01:53:25.519752 3603 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 1 index: 239. (index mismatch)
I20240502 01:53:25.520377 4859 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20240502 01:53:25.597532 2383 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 1 index: 240. (index mismatch)
I20240502 01:53:25.604631 4859 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20240502 01:53:25.608162 2645 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 1 index: 240. (index mismatch)
I20240502 01:53:25.609539 2253 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 238. Preceding OpId from leader: term: 1 index: 240. (index mismatch)
I20240502 01:53:25.614673 4857 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20240502 01:53:25.615350 4857 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 239, Last known committed idx: 238, Time since last communication: 0.000s
I20240502 01:53:25.627813 4825 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [term 1 LEADER]: Committing config change with OpId 1.239: config changed from index 235 to 239, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:25.638913 1836 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4abedcf755774255867d46154f8131c7 with cas_config_opid_index 235: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:25.642271 2647 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.239: config changed from index 235 to 239, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:25.643436 1841 catalog_manager.cc:5484] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 reported cstate change: config changed from index 235 to 239, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New cstate: current_term: 1 leader_uuid: "58dae5dc94444d5e92e6907c5b637879" committed_config { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:25.645486 2248 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.239: config changed from index 235 to 239, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:25.647622 2383 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.239: config changed from index 235 to 239, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:25.651208 2103 tablet_service.cc:1514] Processing DeleteTablet for tablet 4abedcf755774255867d46154f8131c7 with delete_type TABLET_DATA_TOMBSTONED (TS 269c9061e1a1435e92f79f0b9486dfea not found in new config with opid_index 239) from {username='slave'} at 127.0.0.1:50686
I20240502 01:53:25.706317 3602 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.239: config changed from index 235 to 239, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New config: { opid_index: 239 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } }
I20240502 01:53:25.820626 4878 tablet_replica.cc:330] T 4abedcf755774255867d46154f8131c7 P 269c9061e1a1435e92f79f0b9486dfea: stopping tablet replica
I20240502 01:53:25.820926 4878 raft_consensus.cc:2229] T 4abedcf755774255867d46154f8131c7 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:25.824000 4878 raft_consensus.cc:2258] T 4abedcf755774255867d46154f8131c7 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:25.825803 4878 ts_tablet_manager.cc:1875] T 4abedcf755774255867d46154f8131c7 P 269c9061e1a1435e92f79f0b9486dfea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:25.868117 4878 ts_tablet_manager.cc:1888] T 4abedcf755774255867d46154f8131c7 P 269c9061e1a1435e92f79f0b9486dfea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.238
I20240502 01:53:25.872956 4878 log.cc:1196] T 4abedcf755774255867d46154f8131c7 P 269c9061e1a1435e92f79f0b9486dfea: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/wals/4abedcf755774255867d46154f8131c7
I20240502 01:53:25.874028 1837 catalog_manager.cc:4830] TS 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621): tablet 4abedcf755774255867d46154f8131c7 (table rebalance_test_table_0 [id=9b1fbaeec6af4ae5a31888af58780907]) successfully deleted
W20240502 01:53:26.010718 2453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.199:46381, user_credentials={real_user=slave}} blocked reactor thread for 63123.8us
W20240502 01:53:26.925484 2189 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 121487us
I20240502 01:53:27.240772 2103 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:27.267060 3585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:27.290416 2496 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:53:27.294988 2453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 98636.4us
W20240502 01:53:27.314682 2189 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 84124us
W20240502 01:53:27.393473 2188 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.196:39225, user_credentials={real_user=slave}} blocked reactor thread for 44857.6us
I20240502 01:53:27.478545 1971 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:27.514497 2630 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:53:27.572481 2322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 47404.3us
I20240502 01:53:27.593777 2366 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:53:27.740927 2319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.197:38987, user_credentials={real_user=slave}} blocked reactor thread for 218092us
I20240502 01:53:27.976045 2234 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 01:53:27.978422 2453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 119588us
W20240502 01:53:28.132107 2451 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.195:39353, user_credentials={real_user=slave}} blocked reactor thread for 78540.1us
W20240502 01:53:29.229800 2451 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.198:37585, user_credentials={real_user=slave}} blocked reactor thread for 115038us
W20240502 01:53:29.236124 2189 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 116612us
W20240502 01:53:29.351964 2453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 252134us
W20240502 01:53:29.464910 2451 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.195:39353, user_credentials={real_user=slave}} blocked reactor thread for 44154.3us
I20240502 01:53:29.891132 3585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:53:30.210335 2056 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.198:37585, user_credentials={real_user=slave}} blocked reactor thread for 183292us
I20240502 01:53:30.291057 2103 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:53:30.345965 2322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 57597.5us
I20240502 01:53:30.417165 2234 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:30.542508 1971 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:30.581457 2366 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:30.658123 2496 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:30.671288 2630 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 01:53:31.240373 2453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 236304us
W20240502 01:53:31.496248 2452 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.196:39225, user_credentials={real_user=slave}} blocked reactor thread for 70381.9us
W20240502 01:53:31.677238 2319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.197:38987, user_credentials={real_user=slave}} blocked reactor thread for 77471.6us
W20240502 01:53:32.919868 2322 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 53611.1us
W20240502 01:53:33.527980 2319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.195:39353, user_credentials={real_user=slave}} blocked reactor thread for 111724us
W20240502 01:53:33.778007 2453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 44494.3us
W20240502 01:53:34.340034 2454 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 70791.2us
W20240502 01:53:34.386307 2453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 45340.7us
W20240502 01:53:34.433629 2453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 45164.6us
W20240502 01:53:34.723817 2319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.195:39353, user_credentials={real_user=slave}} blocked reactor thread for 86752.3us
W20240502 01:53:35.043439 2453 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 54356.7us
I20240502 01:53:35.513057 3585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:53:35.601006 2454 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 172340us
I20240502 01:53:35.696502 2103 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:35.717433 2234 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:35.896729 2366 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 01:53:35.943250 2451 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.198:37585, user_credentials={real_user=slave}} blocked reactor thread for 46668.6us
I20240502 01:53:36.212563 2630 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:36.303232 2496 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:36.340889 1971 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:53:36.404181 3827 log.cc:924] Time spent T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879: Append to log took a long time: real 0.061s user 0.012s sys 0.000s
W20240502 01:53:36.470834 2451 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.198:37585, user_credentials={real_user=slave}} blocked reactor thread for 54556.8us
W20240502 01:53:36.477977 1925 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.195:39353, user_credentials={real_user=slave}} blocked reactor thread for 62130.8us
W20240502 01:53:36.597158 2321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 179390us
W20240502 01:53:37.814965 2319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.195:39353, user_credentials={real_user=slave}} blocked reactor thread for 51501.8us
I20240502 01:53:38.652633 3585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:38.754571 2103 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:38.904470 2630 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:38.973035 2366 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:39.063644 1971 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:39.155026 2234 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:39.596702 2496 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:53:39.851258 2454 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:39621, user_credentials={real_user=slave}} blocked reactor thread for 50091.5us
W20240502 01:53:39.852399 2189 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:39307, user_credentials={real_user=slave}} blocked reactor thread for 48834.9us
W20240502 01:53:40.173485 2321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.199:46381, user_credentials={real_user=slave}} blocked reactor thread for 48643.9us
W20240502 01:53:40.497175 2189 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.199:46381, user_credentials={real_user=slave}} blocked reactor thread for 63407.3us
I20240502 01:53:40.715111 2103 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:40.720118 3585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:40.720360 2234 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:40.741334 2496 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:40.761505 1971 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:40.803339 2366 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:40.821681 2630 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
4689208abb42472a9ef49d5db3ba5cda | 127.31.161.254:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.31.161.212:35723 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
163649b5fbb140f18dccdf75f608b0f9 | 127.31.161.199:46381 | HEALTHY | <none> | 0 | 0
269c9061e1a1435e92f79f0b9486dfea | 127.31.161.194:39621 | HEALTHY | <none> | 1 | 0
446c7f9bf31a4acc9dae8daf43330906 | 127.31.161.195:39353 | HEALTHY | <none> | 2 | 0
4d99e1fdda704211b3d101f892c57119 | 127.31.161.197:38987 | HEALTHY | <none> | 4 | 0
58dae5dc94444d5e92e6907c5b637879 | 127.31.161.193:39307 | HEALTHY | <none> | 1 | 0
7ef43ea75ecb46ee905b781c91aed95a | 127.31.161.196:39225 | HEALTHY | <none> | 5 | 0
9a3448a425c041258991dc60dd64ae7d | 127.31.161.198:37585 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.31.161.193 | experimental | 127.31.161.193:39307
local_ip_for_outbound_sockets | 127.31.161.194 | experimental | 127.31.161.194:39621
local_ip_for_outbound_sockets | 127.31.161.195 | experimental | 127.31.161.195:39353
local_ip_for_outbound_sockets | 127.31.161.196 | experimental | 127.31.161.196:39225
local_ip_for_outbound_sockets | 127.31.161.197 | experimental | 127.31.161.197:38987
local_ip_for_outbound_sockets | 127.31.161.198 | experimental | 127.31.161.198:37585
local_ip_for_outbound_sockets | 127.31.161.199 | experimental | 127.31.161.199:46381
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.31.161.193:39307
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.31.161.194:39621
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.31.161.195:39353
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.31.161.196:39225
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.31.161.197:38987
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.31.161.198:37585
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.31.161.199:46381
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.31.161.212:35723 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 11
Median | 11
Third Quartile | 12
Maximum | 12
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 80
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 01:53:40.894310 32391 log_verifier.cc:126] Checking tablet 0225b684df4f41499f76fe10b72a8861
I20240502 01:53:41.195955 32391 log_verifier.cc:177] Verified matching terms for 283 ops in tablet 0225b684df4f41499f76fe10b72a8861
I20240502 01:53:41.196722 32391 log_verifier.cc:126] Checking tablet 1983e0a53ee54b62afa16676bf986332
I20240502 01:53:41.544453 32391 log_verifier.cc:177] Verified matching terms for 350 ops in tablet 1983e0a53ee54b62afa16676bf986332
I20240502 01:53:41.545480 32391 log_verifier.cc:126] Checking tablet 27398ad1eb7f4bfab185f8affe1f790d
I20240502 01:53:41.865845 32391 log_verifier.cc:177] Verified matching terms for 322 ops in tablet 27398ad1eb7f4bfab185f8affe1f790d
I20240502 01:53:41.866775 32391 log_verifier.cc:126] Checking tablet 341e0200ea654dcfb91133135b458a16
I20240502 01:53:42.246917 32391 log_verifier.cc:177] Verified matching terms for 379 ops in tablet 341e0200ea654dcfb91133135b458a16
I20240502 01:53:42.248045 32391 log_verifier.cc:126] Checking tablet 4068710224254825bf42671418592ece
I20240502 01:53:42.537389 32391 log_verifier.cc:177] Verified matching terms for 267 ops in tablet 4068710224254825bf42671418592ece
I20240502 01:53:42.538112 32391 log_verifier.cc:126] Checking tablet 4220d1a63bd74ad0a5cac5a2f1bb988c
I20240502 01:53:42.888098 32391 log_verifier.cc:177] Verified matching terms for 348 ops in tablet 4220d1a63bd74ad0a5cac5a2f1bb988c
I20240502 01:53:42.888969 32391 log_verifier.cc:126] Checking tablet 4abedcf755774255867d46154f8131c7
I20240502 01:53:43.260483 32391 log_verifier.cc:177] Verified matching terms for 372 ops in tablet 4abedcf755774255867d46154f8131c7
I20240502 01:53:43.261415 32391 log_verifier.cc:126] Checking tablet 677599c7914a4bafb83995cf3240679d
I20240502 01:53:43.537489 32391 log_verifier.cc:177] Verified matching terms for 266 ops in tablet 677599c7914a4bafb83995cf3240679d
I20240502 01:53:43.538179 32391 log_verifier.cc:126] Checking tablet 73f01fd0dc7f488ca215b7f95ea8d988
I20240502 01:53:43.910900 32391 log_verifier.cc:177] Verified matching terms for 353 ops in tablet 73f01fd0dc7f488ca215b7f95ea8d988
I20240502 01:53:43.911819 32391 log_verifier.cc:126] Checking tablet 74753a745d874fcea0fa9dfc4649aa14
I20240502 01:53:43.965893 32391 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 74753a745d874fcea0fa9dfc4649aa14
I20240502 01:53:43.965998 32391 log_verifier.cc:126] Checking tablet 9895c93b27764c22b77087d74e0149c6
I20240502 01:53:44.317059 32391 log_verifier.cc:177] Verified matching terms for 337 ops in tablet 9895c93b27764c22b77087d74e0149c6
I20240502 01:53:44.317961 32391 log_verifier.cc:126] Checking tablet c291b2a5e9d140f881b2bd54f8c9ed19
I20240502 01:53:44.666138 32391 log_verifier.cc:177] Verified matching terms for 316 ops in tablet c291b2a5e9d140f881b2bd54f8c9ed19
I20240502 01:53:44.667047 32391 log_verifier.cc:126] Checking tablet d592edf3298942cd93739aa846f867a4
I20240502 01:53:45.010823 32391 log_verifier.cc:177] Verified matching terms for 327 ops in tablet d592edf3298942cd93739aa846f867a4
I20240502 01:53:45.011633 32391 log_verifier.cc:126] Checking tablet e4672bc55a874f159bf10855f0698ba1
I20240502 01:53:45.360170 32391 log_verifier.cc:177] Verified matching terms for 345 ops in tablet e4672bc55a874f159bf10855f0698ba1
I20240502 01:53:45.361032 32391 log_verifier.cc:126] Checking tablet e567c877188549fc965637e89bf437ae
I20240502 01:53:45.689149 32391 log_verifier.cc:177] Verified matching terms for 301 ops in tablet e567c877188549fc965637e89bf437ae
I20240502 01:53:45.689891 32391 log_verifier.cc:126] Checking tablet f4a07f0fe1964f958ad7ca4a63d054a6
I20240502 01:53:46.018148 32391 log_verifier.cc:177] Verified matching terms for 331 ops in tablet f4a07f0fe1964f958ad7ca4a63d054a6
I20240502 01:53:46.021823 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.200:0
--local_ip_for_outbound_sockets=127.31.161.200
--webserver_interface=127.31.161.200
--webserver_port=0
--tserver_master_addrs=127.31.161.254:45525
--builtin_ntp_servers=127.31.161.212:35723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:53:46.387866 5442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:53:46.388177 5442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:53:46.388284 5442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:53:46.396693 5442 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:53:46.396811 5442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:53:46.396934 5442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.200
I20240502 01:53:46.406725 5442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.31.161.200
--webserver_port=0
--tserver_master_addrs=127.31.161.254:45525
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.200
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:53:46.408129 5442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:53:46.410180 5442 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:53:46.423002 5449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:46.423381 5448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:53:46.423848 5451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:53:46.424090 5442 server_base.cc:1034] running on GCE node
I20240502 01:53:46.424669 5442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:53:46.425707 5442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:53:46.426899 5442 hybrid_clock.cc:648] HybridClock initialized: now 1714614826426844 us; error 47 us; skew 500 ppm
I20240502 01:53:46.427234 5442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:53:46.434338 5442 webserver.cc:460] Webserver started at http://127.31.161.200:35031/ using document root <none> and password file <none>
I20240502 01:53:46.435081 5442 fs_manager.cc:359] Metadata directory not provided
I20240502 01:53:46.435196 5442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:53:46.435497 5442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:53:46.438156 5442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "9715934145414f2496a4105e95aa0987"
format_stamp: "Formatted at 2024-05-02 01:53:46 on dist-test-slave-06l8"
I20240502 01:53:46.438889 5442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "9715934145414f2496a4105e95aa0987"
format_stamp: "Formatted at 2024-05-02 01:53:46 on dist-test-slave-06l8"
I20240502 01:53:46.444262 5442 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 01:53:46.447950 5457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:46.449517 5442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240502 01:53:46.449671 5442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "9715934145414f2496a4105e95aa0987"
format_stamp: "Formatted at 2024-05-02 01:53:46 on dist-test-slave-06l8"
I20240502 01:53:46.449882 5442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:53:46.473210 5442 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:53:46.474179 5442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:53:46.474507 5442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:53:46.475837 5442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:53:46.477526 5442 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:53:46.477615 5442 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:46.477721 5442 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:53:46.477790 5442 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:53:46.535944 5442 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.200:36271
I20240502 01:53:46.536001 5570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.200:36271 every 8 connection(s)
I20240502 01:53:46.537336 5442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:53:46.547139 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 5442
I20240502 01:53:46.547334 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 01:53:46.550510 5571 heartbeater.cc:347] Connected to a master server at 127.31.161.254:45525
I20240502 01:53:46.550827 5571 heartbeater.cc:464] Registering TS with master...
I20240502 01:53:46.551642 5571 heartbeater.cc:511] Master 127.31.161.254:45525 requested a full tablet report, sending...
I20240502 01:53:46.553117 1841 ts_manager.cc:194] Registered new tserver with Master: 9715934145414f2496a4105e95aa0987 (127.31.161.200:36271)
I20240502 01:53:46.554178 1841 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.200:50911
I20240502 01:53:47.127676 2366 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:47.148983 2103 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:47.156332 2496 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:47.182091 3585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:47.196480 5505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:47.208309 1971 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 01:53:47.212946 2187 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.197:38987, user_credentials={real_user=slave}} blocked reactor thread for 70442.7us
I20240502 01:53:47.215788 2234 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:47.242595 2630 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:47.494230 2103 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:47.498898 3585 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:47.509696 2234 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:47.522985 1971 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:47.550571 2496 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:47.553133 2366 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:53:47.556650 5571 heartbeater.cc:503] Master 127.31.161.254:45525 was elected leader, sending a full tablet report...
I20240502 01:53:47.604768 5505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:47.612795 2630 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:47.686453 2516 consensus_queue.cc:237] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 345, Committed index: 345, Last appended: 1.345, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:47.689612 2254 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 345. Preceding OpId from leader: term: 1 index: 346. (index mismatch)
I20240502 01:53:47.690160 5437 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 346, Last known committed idx: 345, Time since last communication: 0.000s
I20240502 01:53:47.703732 2117 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 345. Preceding OpId from leader: term: 1 index: 346. (index mismatch)
I20240502 01:53:47.703812 1985 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 345. Preceding OpId from leader: term: 1 index: 346. (index mismatch)
I20240502 01:53:47.704689 2385 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 345. Preceding OpId from leader: term: 1 index: 346. (index mismatch)
I20240502 01:53:47.704811 5437 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 346, Last known committed idx: 345, Time since last communication: 0.000s
I20240502 01:53:47.705273 5668 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 346, Last known committed idx: 345, Time since last communication: 0.000s
I20240502 01:53:47.708314 5437 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 346, Last known committed idx: 345, Time since last communication: 0.000s
W20240502 01:53:47.709275 2454 consensus_peers.cc:485] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 -> Peer 9715934145414f2496a4105e95aa0987 (127.31.161.200:36271): Couldn't send request to peer 9715934145414f2496a4105e95aa0987. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4672bc55a874f159bf10855f0698ba1. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:47.709623 5668 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [term 1 LEADER]: Committing config change with OpId 1.346: config changed from index 103 to 346, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.710443 1985 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.346: config changed from index 103 to 346, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.718281 1847 catalog_manager.cc:5484] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879 reported cstate change: config changed from index 103 to 346, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "4d99e1fdda704211b3d101f892c57119" committed_config { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.718755 2254 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.346: config changed from index 103 to 346, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.720083 2385 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.346: config changed from index 103 to 346, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.729491 2117 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.346: config changed from index 103 to 346, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.735890 2647 consensus_queue.cc:237] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 301, Committed index: 301, Last appended: 1.301, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:47.743311 3604 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 301. Preceding OpId from leader: term: 1 index: 302. (index mismatch)
I20240502 01:53:47.743672 2117 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 301. Preceding OpId from leader: term: 1 index: 302. (index mismatch)
I20240502 01:53:47.744184 5626 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s
I20240502 01:53:47.744449 5439 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s
I20240502 01:53:47.745724 2254 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 301. Preceding OpId from leader: term: 1 index: 302. (index mismatch)
I20240502 01:53:47.746228 5687 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s
I20240502 01:53:47.747607 2385 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 301. Preceding OpId from leader: term: 1 index: 302. (index mismatch)
I20240502 01:53:47.748270 5626 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 302, Last known committed idx: 301, Time since last communication: 0.000s
W20240502 01:53:47.750789 2586 consensus_peers.cc:485] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d -> Peer 9715934145414f2496a4105e95aa0987 (127.31.161.200:36271): Couldn't send request to peer 9715934145414f2496a4105e95aa0987. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e567c877188549fc965637e89bf437ae. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:47.751948 5439 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [term 1 LEADER]: Committing config change with OpId 1.302: config changed from index 80 to 302, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.752451 3604 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.302: config changed from index 80 to 302, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.752744 2385 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.302: config changed from index 80 to 302, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.757380 2254 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.302: config changed from index 80 to 302, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.758782 1841 catalog_manager.cc:5484] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a reported cstate change: config changed from index 80 to 302, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "9a3448a425c041258991dc60dd64ae7d" committed_config { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.762425 2117 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.302: config changed from index 80 to 302, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.782795 2385 consensus_queue.cc:237] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 266, Committed index: 266, Last appended: 1.266, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:47.785878 2254 raft_consensus.cc:1261] T 677599c7914a4bafb83995cf3240679d P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 266. Preceding OpId from leader: term: 1 index: 267. (index mismatch)
I20240502 01:53:47.786403 1985 raft_consensus.cc:1261] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 266. Preceding OpId from leader: term: 1 index: 267. (index mismatch)
I20240502 01:53:47.787060 2117 raft_consensus.cc:1261] T 677599c7914a4bafb83995cf3240679d P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 266. Preceding OpId from leader: term: 1 index: 267. (index mismatch)
I20240502 01:53:47.787299 2647 raft_consensus.cc:1261] T 677599c7914a4bafb83995cf3240679d P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 266. Preceding OpId from leader: term: 1 index: 267. (index mismatch)
I20240502 01:53:47.787292 5577 consensus_queue.cc:1035] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 266, Time since last communication: 0.000s
I20240502 01:53:47.787845 5577 consensus_queue.cc:1035] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 266, Time since last communication: 0.000s
I20240502 01:53:47.788319 5577 consensus_queue.cc:1035] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 266, Time since last communication: 0.000s
I20240502 01:53:47.790638 5577 consensus_queue.cc:1035] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 267, Last known committed idx: 266, Time since last communication: 0.000s
W20240502 01:53:47.802572 2322 consensus_peers.cc:485] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a -> Peer 9715934145414f2496a4105e95aa0987 (127.31.161.200:36271): Couldn't send request to peer 9715934145414f2496a4105e95aa0987. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 677599c7914a4bafb83995cf3240679d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:47.809947 5577 raft_consensus.cc:2932] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Committing config change with OpId 1.267: config changed from index -1 to 267, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.810853 2254 raft_consensus.cc:2932] T 677599c7914a4bafb83995cf3240679d P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.267: config changed from index -1 to 267, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.811889 1985 raft_consensus.cc:2932] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.267: config changed from index -1 to 267, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.812062 2647 raft_consensus.cc:2932] T 677599c7914a4bafb83995cf3240679d P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.267: config changed from index -1 to 267, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.814692 2117 raft_consensus.cc:2932] T 677599c7914a4bafb83995cf3240679d P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.267: config changed from index -1 to 267, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.819479 1841 catalog_manager.cc:5484] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879 reported cstate change: config changed from index -1 to 267, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "7ef43ea75ecb46ee905b781c91aed95a" committed_config { opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.834095 2254 consensus_queue.cc:237] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 327, Committed index: 327, Last appended: 1.327, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:47.836854 3604 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 327. Preceding OpId from leader: term: 1 index: 328. (index mismatch)
I20240502 01:53:47.837217 1985 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 327. Preceding OpId from leader: term: 1 index: 328. (index mismatch)
I20240502 01:53:47.837705 5435 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 328, Last known committed idx: 327, Time since last communication: 0.000s
I20240502 01:53:47.838284 5435 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 328, Last known committed idx: 327, Time since last communication: 0.000s
I20240502 01:53:47.838599 2385 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 327. Preceding OpId from leader: term: 1 index: 328. (index mismatch)
I20240502 01:53:47.839084 5435 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 328, Last known committed idx: 327, Time since last communication: 0.000s
I20240502 01:53:47.839313 2516 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 327. Preceding OpId from leader: term: 1 index: 328. (index mismatch)
I20240502 01:53:47.839943 5435 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 328, Last known committed idx: 327, Time since last communication: 0.000s
W20240502 01:53:47.860603 2190 consensus_peers.cc:485] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 -> Peer 9715934145414f2496a4105e95aa0987 (127.31.161.200:36271): Couldn't send request to peer 9715934145414f2496a4105e95aa0987. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d592edf3298942cd93739aa846f867a4. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:47.869167 5435 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEADER]: Committing config change with OpId 1.328: config changed from index 125 to 328, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.870023 2385 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.328: config changed from index 125 to 328, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.870707 3604 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.328: config changed from index 125 to 328, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.874200 1985 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.328: config changed from index 125 to 328, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.875025 2516 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.328: config changed from index 125 to 328, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.878484 1849 catalog_manager.cc:5484] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 reported cstate change: config changed from index 125 to 328, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "446c7f9bf31a4acc9dae8daf43330906" committed_config { opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:47.905511 2117 consensus_queue.cc:237] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:47.908370 2647 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:47.909116 4609 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:53:47.916599 2059 consensus_peers.cc:485] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea -> Peer 9715934145414f2496a4105e95aa0987 (127.31.161.200:36271): Couldn't send request to peer 9715934145414f2496a4105e95aa0987. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74753a745d874fcea0fa9dfc4649aa14. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:47.922210 2516 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:47.922410 3604 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:47.922423 2385 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:53:47.922909 4609 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:47.923218 5730 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:47.923686 4609 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:53:47.927807 4609 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.928459 2385 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.931859 2516 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.933446 1841 catalog_manager.cc:5484] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea reported cstate change: config changed from index 4 to 5, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "269c9061e1a1435e92f79f0b9486dfea" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:47.939461 2647 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:47.949604 3604 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:48.233983 5783 ts_tablet_manager.cc:902] T e4672bc55a874f159bf10855f0698ba1 P 9715934145414f2496a4105e95aa0987: Initiating tablet copy from peer 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987)
I20240502 01:53:48.236035 5783 tablet_copy_client.cc:308] T e4672bc55a874f159bf10855f0698ba1 P 9715934145414f2496a4105e95aa0987: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.197:38987
I20240502 01:53:48.255465 2539 tablet_copy_service.cc:140] P 4d99e1fdda704211b3d101f892c57119: Received BeginTabletCopySession request for tablet e4672bc55a874f159bf10855f0698ba1 from peer 9715934145414f2496a4105e95aa0987 ({username='slave'} at 127.31.161.200:50693)
I20240502 01:53:48.255626 2539 tablet_copy_service.cc:161] P 4d99e1fdda704211b3d101f892c57119: Beginning new tablet copy session on tablet e4672bc55a874f159bf10855f0698ba1 from peer 9715934145414f2496a4105e95aa0987 at {username='slave'} at 127.31.161.200:50693: session id = 9715934145414f2496a4105e95aa0987-e4672bc55a874f159bf10855f0698ba1
I20240502 01:53:48.258057 2539 tablet_copy_source_session.cc:205] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:48.263463 5783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4672bc55a874f159bf10855f0698ba1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.273842 5783 tablet_copy_client.cc:783] T e4672bc55a874f159bf10855f0698ba1 P 9715934145414f2496a4105e95aa0987: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:48.274168 5783 tablet_copy_client.cc:647] T e4672bc55a874f159bf10855f0698ba1 P 9715934145414f2496a4105e95aa0987: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:48.277002 5783 tablet_copy_client.cc:523] T e4672bc55a874f159bf10855f0698ba1 P 9715934145414f2496a4105e95aa0987: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:48.280230 5783 tablet_bootstrap.cc:492] T e4672bc55a874f159bf10855f0698ba1 P 9715934145414f2496a4105e95aa0987: Bootstrap starting.
I20240502 01:53:48.330405 5783 log.cc:824] T e4672bc55a874f159bf10855f0698ba1 P 9715934145414f2496a4105e95aa0987: Log is configured to *not* fsync() on all Append() calls
I20240502 01:53:48.354184 5798 ts_tablet_manager.cc:902] T e567c877188549fc965637e89bf437ae P 9715934145414f2496a4105e95aa0987: Initiating tablet copy from peer 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585)
I20240502 01:53:48.356878 5802 ts_tablet_manager.cc:902] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987: Initiating tablet copy from peer 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353)
I20240502 01:53:48.364194 5798 tablet_copy_client.cc:308] T e567c877188549fc965637e89bf437ae P 9715934145414f2496a4105e95aa0987: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.198:37585
I20240502 01:53:48.371960 5802 tablet_copy_client.cc:308] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:39353
I20240502 01:53:48.372648 2670 tablet_copy_service.cc:140] P 9a3448a425c041258991dc60dd64ae7d: Received BeginTabletCopySession request for tablet e567c877188549fc965637e89bf437ae from peer 9715934145414f2496a4105e95aa0987 ({username='slave'} at 127.31.161.200:53303)
I20240502 01:53:48.372803 2670 tablet_copy_service.cc:161] P 9a3448a425c041258991dc60dd64ae7d: Beginning new tablet copy session on tablet e567c877188549fc965637e89bf437ae from peer 9715934145414f2496a4105e95aa0987 at {username='slave'} at 127.31.161.200:53303: session id = 9715934145414f2496a4105e95aa0987-e567c877188549fc965637e89bf437ae
I20240502 01:53:48.375815 2670 tablet_copy_source_session.cc:205] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:48.377522 5798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e567c877188549fc965637e89bf437ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.379738 2275 tablet_copy_service.cc:140] P 446c7f9bf31a4acc9dae8daf43330906: Received BeginTabletCopySession request for tablet d592edf3298942cd93739aa846f867a4 from peer 9715934145414f2496a4105e95aa0987 ({username='slave'} at 127.31.161.200:43913)
I20240502 01:53:48.379922 2275 tablet_copy_service.cc:161] P 446c7f9bf31a4acc9dae8daf43330906: Beginning new tablet copy session on tablet d592edf3298942cd93739aa846f867a4 from peer 9715934145414f2496a4105e95aa0987 at {username='slave'} at 127.31.161.200:43913: session id = 9715934145414f2496a4105e95aa0987-d592edf3298942cd93739aa846f867a4
I20240502 01:53:48.382812 2275 tablet_copy_source_session.cc:205] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:48.383570 5798 tablet_copy_client.cc:783] T e567c877188549fc965637e89bf437ae P 9715934145414f2496a4105e95aa0987: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:48.383831 5798 tablet_copy_client.cc:647] T e567c877188549fc965637e89bf437ae P 9715934145414f2496a4105e95aa0987: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:48.384339 5802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d592edf3298942cd93739aa846f867a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.387279 5798 tablet_copy_client.cc:523] T e567c877188549fc965637e89bf437ae P 9715934145414f2496a4105e95aa0987: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:48.390116 5798 tablet_bootstrap.cc:492] T e567c877188549fc965637e89bf437ae P 9715934145414f2496a4105e95aa0987: Bootstrap starting.
I20240502 01:53:48.391734 5802 tablet_copy_client.cc:783] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:48.392014 5802 tablet_copy_client.cc:647] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:48.398330 5802 tablet_copy_client.cc:523] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:48.402060 5802 tablet_bootstrap.cc:492] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987: Bootstrap starting.
I20240502 01:53:48.417642 5811 ts_tablet_manager.cc:902] T 677599c7914a4bafb83995cf3240679d P 9715934145414f2496a4105e95aa0987: Initiating tablet copy from peer 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225)
I20240502 01:53:48.429622 5810 ts_tablet_manager.cc:902] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987: Initiating tablet copy from peer 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621)
I20240502 01:53:48.459242 5811 tablet_copy_client.cc:308] T 677599c7914a4bafb83995cf3240679d P 9715934145414f2496a4105e95aa0987: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:39225
I20240502 01:53:48.473390 2408 tablet_copy_service.cc:140] P 7ef43ea75ecb46ee905b781c91aed95a: Received BeginTabletCopySession request for tablet 677599c7914a4bafb83995cf3240679d from peer 9715934145414f2496a4105e95aa0987 ({username='slave'} at 127.31.161.200:50229)
I20240502 01:53:48.473588 2408 tablet_copy_service.cc:161] P 7ef43ea75ecb46ee905b781c91aed95a: Beginning new tablet copy session on tablet 677599c7914a4bafb83995cf3240679d from peer 9715934145414f2496a4105e95aa0987 at {username='slave'} at 127.31.161.200:50229: session id = 9715934145414f2496a4105e95aa0987-677599c7914a4bafb83995cf3240679d
I20240502 01:53:48.476706 2408 tablet_copy_source_session.cc:205] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:48.477835 5810 tablet_copy_client.cc:308] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.194:39621
I20240502 01:53:48.478286 5811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 677599c7914a4bafb83995cf3240679d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.493978 2144 tablet_copy_service.cc:140] P 269c9061e1a1435e92f79f0b9486dfea: Received BeginTabletCopySession request for tablet 74753a745d874fcea0fa9dfc4649aa14 from peer 9715934145414f2496a4105e95aa0987 ({username='slave'} at 127.31.161.200:52399)
I20240502 01:53:48.494163 2144 tablet_copy_service.cc:161] P 269c9061e1a1435e92f79f0b9486dfea: Beginning new tablet copy session on tablet 74753a745d874fcea0fa9dfc4649aa14 from peer 9715934145414f2496a4105e95aa0987 at {username='slave'} at 127.31.161.200:52399: session id = 9715934145414f2496a4105e95aa0987-74753a745d874fcea0fa9dfc4649aa14
I20240502 01:53:48.494923 5811 tablet_copy_client.cc:783] T 677599c7914a4bafb83995cf3240679d P 9715934145414f2496a4105e95aa0987: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:48.495203 5811 tablet_copy_client.cc:647] T 677599c7914a4bafb83995cf3240679d P 9715934145414f2496a4105e95aa0987: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:48.497148 2144 tablet_copy_source_session.cc:205] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:48.498673 5810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74753a745d874fcea0fa9dfc4649aa14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:48.502401 5811 tablet_copy_client.cc:523] T 677599c7914a4bafb83995cf3240679d P 9715934145414f2496a4105e95aa0987: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:48.505438 5810 tablet_copy_client.cc:783] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:48.505704 5810 tablet_copy_client.cc:647] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:48.506549 5811 tablet_bootstrap.cc:492] T 677599c7914a4bafb83995cf3240679d P 9715934145414f2496a4105e95aa0987: Bootstrap starting.
I20240502 01:53:48.507658 5810 tablet_copy_client.cc:523] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:48.523308 5810 tablet_bootstrap.cc:492] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987: Bootstrap starting.
I20240502 01:53:48.575129 5810 tablet_bootstrap.cc:492] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:48.575800 5810 tablet_bootstrap.cc:492] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987: Bootstrap complete.
I20240502 01:53:48.576364 5810 ts_tablet_manager.cc:1367] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987: Time spent bootstrapping tablet: real 0.053s user 0.007s sys 0.002s
I20240502 01:53:48.599145 5810 raft_consensus.cc:348] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:48.601204 5810 raft_consensus.cc:729] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9715934145414f2496a4105e95aa0987, State: Initialized, Role: LEARNER
I20240502 01:53:48.601919 5810 consensus_queue.cc:260] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:48.623452 5810 ts_tablet_manager.cc:1398] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987: Time spent starting tablet: real 0.047s user 0.009s sys 0.007s
I20240502 01:53:48.624676 2144 tablet_copy_service.cc:342] P 269c9061e1a1435e92f79f0b9486dfea: Request end of tablet copy session 9715934145414f2496a4105e95aa0987-74753a745d874fcea0fa9dfc4649aa14 received from {username='slave'} at 127.31.161.200:52399
I20240502 01:53:48.624815 2144 tablet_copy_service.cc:433] P 269c9061e1a1435e92f79f0b9486dfea: ending tablet copy session 9715934145414f2496a4105e95aa0987-74753a745d874fcea0fa9dfc4649aa14 on tablet 74753a745d874fcea0fa9dfc4649aa14 with peer 9715934145414f2496a4105e95aa0987
W20240502 01:53:48.790382 2187 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.198:37585, user_credentials={real_user=slave}} blocked reactor thread for 44208.2us
I20240502 01:53:48.863999 5783 tablet_bootstrap.cc:492] T e4672bc55a874f159bf10855f0698ba1 P 9715934145414f2496a4105e95aa0987: Bootstrap replayed 1/1 log segments. Stats: ops{read=346 overwritten=0 applied=346 ignored=0} inserts{seen=1015 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:48.864707 5783 tablet_bootstrap.cc:492] T e4672bc55a874f159bf10855f0698ba1 P 9715934145414f2496a4105e95aa0987: Bootstrap complete.
I20240502 01:53:48.865134 5783 ts_tablet_manager.cc:1367] T e4672bc55a874f159bf10855f0698ba1 P 9715934145414f2496a4105e95aa0987: Time spent bootstrapping tablet: real 0.585s user 0.423s sys 0.033s
I20240502 01:53:48.866863 5783 raft_consensus.cc:348] T e4672bc55a874f159bf10855f0698ba1 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:48.867565 5783 raft_consensus.cc:729] T e4672bc55a874f159bf10855f0698ba1 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9715934145414f2496a4105e95aa0987, State: Initialized, Role: LEARNER
I20240502 01:53:48.867842 5783 consensus_queue.cc:260] T e4672bc55a874f159bf10855f0698ba1 P 9715934145414f2496a4105e95aa0987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 346, Last appended: 1.346, Last appended by leader: 346, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 346 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:48.873003 5783 ts_tablet_manager.cc:1398] T e4672bc55a874f159bf10855f0698ba1 P 9715934145414f2496a4105e95aa0987: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:53:48.873937 2539 tablet_copy_service.cc:342] P 4d99e1fdda704211b3d101f892c57119: Request end of tablet copy session 9715934145414f2496a4105e95aa0987-e4672bc55a874f159bf10855f0698ba1 received from {username='slave'} at 127.31.161.200:50693
I20240502 01:53:48.874068 2539 tablet_copy_service.cc:433] P 4d99e1fdda704211b3d101f892c57119: ending tablet copy session 9715934145414f2496a4105e95aa0987-e4672bc55a874f159bf10855f0698ba1 on tablet e4672bc55a874f159bf10855f0698ba1 with peer 9715934145414f2496a4105e95aa0987
I20240502 01:53:48.926506 5525 raft_consensus.cc:1203] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:53:49.065251 5802 tablet_bootstrap.cc:492] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987: Bootstrap replayed 1/1 log segments. Stats: ops{read=328 overwritten=0 applied=328 ignored=0} inserts{seen=997 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:49.065760 5802 tablet_bootstrap.cc:492] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987: Bootstrap complete.
I20240502 01:53:49.066109 5802 ts_tablet_manager.cc:1367] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987: Time spent bootstrapping tablet: real 0.664s user 0.401s sys 0.032s
I20240502 01:53:49.067008 5802 raft_consensus.cc:348] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:49.067340 5802 raft_consensus.cc:729] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9715934145414f2496a4105e95aa0987, State: Initialized, Role: LEARNER
I20240502 01:53:49.067627 5802 consensus_queue.cc:260] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 328, Last appended: 1.328, Last appended by leader: 328, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 328 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:49.068393 5802 ts_tablet_manager.cc:1398] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:49.069429 2275 tablet_copy_service.cc:342] P 446c7f9bf31a4acc9dae8daf43330906: Request end of tablet copy session 9715934145414f2496a4105e95aa0987-d592edf3298942cd93739aa846f867a4 received from {username='slave'} at 127.31.161.200:43913
I20240502 01:53:49.069542 2275 tablet_copy_service.cc:433] P 446c7f9bf31a4acc9dae8daf43330906: ending tablet copy session 9715934145414f2496a4105e95aa0987-d592edf3298942cd93739aa846f867a4 on tablet d592edf3298942cd93739aa846f867a4 with peer 9715934145414f2496a4105e95aa0987
I20240502 01:53:49.075381 5798 tablet_bootstrap.cc:492] T e567c877188549fc965637e89bf437ae P 9715934145414f2496a4105e95aa0987: Bootstrap replayed 1/1 log segments. Stats: ops{read=302 overwritten=0 applied=302 ignored=0} inserts{seen=930 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:49.075809 5798 tablet_bootstrap.cc:492] T e567c877188549fc965637e89bf437ae P 9715934145414f2496a4105e95aa0987: Bootstrap complete.
I20240502 01:53:49.076164 5798 ts_tablet_manager.cc:1367] T e567c877188549fc965637e89bf437ae P 9715934145414f2496a4105e95aa0987: Time spent bootstrapping tablet: real 0.686s user 0.346s sys 0.020s
I20240502 01:53:49.076930 5798 raft_consensus.cc:348] T e567c877188549fc965637e89bf437ae P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:49.077311 5798 raft_consensus.cc:729] T e567c877188549fc965637e89bf437ae P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9715934145414f2496a4105e95aa0987, State: Initialized, Role: LEARNER
I20240502 01:53:49.077541 5798 consensus_queue.cc:260] T e567c877188549fc965637e89bf437ae P 9715934145414f2496a4105e95aa0987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 302, Last appended: 1.302, Last appended by leader: 302, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 302 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:49.078155 5798 ts_tablet_manager.cc:1398] T e567c877188549fc965637e89bf437ae P 9715934145414f2496a4105e95aa0987: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:53:49.079005 2670 tablet_copy_service.cc:342] P 9a3448a425c041258991dc60dd64ae7d: Request end of tablet copy session 9715934145414f2496a4105e95aa0987-e567c877188549fc965637e89bf437ae received from {username='slave'} at 127.31.161.200:53303
I20240502 01:53:49.079133 2670 tablet_copy_service.cc:433] P 9a3448a425c041258991dc60dd64ae7d: ending tablet copy session 9715934145414f2496a4105e95aa0987-e567c877188549fc965637e89bf437ae on tablet e567c877188549fc965637e89bf437ae with peer 9715934145414f2496a4105e95aa0987
I20240502 01:53:49.085717 5811 tablet_bootstrap.cc:492] T 677599c7914a4bafb83995cf3240679d P 9715934145414f2496a4105e95aa0987: Bootstrap replayed 1/1 log segments. Stats: ops{read=267 overwritten=0 applied=267 ignored=0} inserts{seen=898 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:49.086100 5811 tablet_bootstrap.cc:492] T 677599c7914a4bafb83995cf3240679d P 9715934145414f2496a4105e95aa0987: Bootstrap complete.
I20240502 01:53:49.086367 5811 ts_tablet_manager.cc:1367] T 677599c7914a4bafb83995cf3240679d P 9715934145414f2496a4105e95aa0987: Time spent bootstrapping tablet: real 0.580s user 0.334s sys 0.025s
I20240502 01:53:49.087086 5811 raft_consensus.cc:348] T 677599c7914a4bafb83995cf3240679d P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:49.087376 5811 raft_consensus.cc:729] T 677599c7914a4bafb83995cf3240679d P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9715934145414f2496a4105e95aa0987, State: Initialized, Role: LEARNER
I20240502 01:53:49.087579 5811 consensus_queue.cc:260] T 677599c7914a4bafb83995cf3240679d P 9715934145414f2496a4105e95aa0987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 267, Last appended: 1.267, Last appended by leader: 267, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 267 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:49.088158 5811 ts_tablet_manager.cc:1398] T 677599c7914a4bafb83995cf3240679d P 9715934145414f2496a4105e95aa0987: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:49.088898 2408 tablet_copy_service.cc:342] P 7ef43ea75ecb46ee905b781c91aed95a: Request end of tablet copy session 9715934145414f2496a4105e95aa0987-677599c7914a4bafb83995cf3240679d received from {username='slave'} at 127.31.161.200:50229
I20240502 01:53:49.089040 2408 tablet_copy_service.cc:433] P 7ef43ea75ecb46ee905b781c91aed95a: ending tablet copy session 9715934145414f2496a4105e95aa0987-677599c7914a4bafb83995cf3240679d on tablet 677599c7914a4bafb83995cf3240679d with peer 9715934145414f2496a4105e95aa0987
I20240502 01:53:49.233757 5525 raft_consensus.cc:1203] T e4672bc55a874f159bf10855f0698ba1 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.345->[1.346-1.346] Dedup: 1.346->[]
I20240502 01:53:49.239498 5525 raft_consensus.cc:1203] T 677599c7914a4bafb83995cf3240679d P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.266->[1.267-1.267] Dedup: 1.267->[]
I20240502 01:53:49.390703 5525 raft_consensus.cc:1203] T e567c877188549fc965637e89bf437ae P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.301->[1.302-1.302] Dedup: 1.302->[]
I20240502 01:53:49.395793 5525 raft_consensus.cc:1203] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.327->[1.328-1.328] Dedup: 1.328->[]
I20240502 01:53:49.462284 5918 raft_consensus.cc:1050] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea: attempting to promote NON_VOTER 9715934145414f2496a4105e95aa0987 to VOTER
I20240502 01:53:49.463330 5918 consensus_queue.cc:237] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:49.466032 2647 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:49.466296 5525 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:49.467272 2385 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:49.467231 5918 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:49.467466 3597 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:49.468031 5918 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:49.468232 2516 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:53:49.469364 5918 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:49.471334 5921 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:49.471691 5920 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:53:49.474157 5920 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.475464 5525 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.475841 2516 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.477708 2385 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.480989 1844 catalog_manager.cc:5484] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987 reported cstate change: config changed from index 5 to 6, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "269c9061e1a1435e92f79f0b9486dfea" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.481591 3597 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.483086 2647 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } attrs { replace: true } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.499440 2117 consensus_queue.cc:237] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:49.501732 5525 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:53:49.502149 3604 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:53:49.502475 2385 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:53:49.502593 5920 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:53:49.502702 2516 raft_consensus.cc:1261] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 269c9061e1a1435e92f79f0b9486dfea: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:53:49.503145 5920 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:53:49.503439 5921 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:53:49.503962 5921 consensus_queue.cc:1035] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:53:49.506322 5921 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 269c9061e1a1435e92f79f0b9486dfea [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9a3448a425c041258991dc60dd64ae7d (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.507297 2516 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9a3448a425c041258991dc60dd64ae7d (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.509058 3604 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9a3448a425c041258991dc60dd64ae7d (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.510538 1837 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 74753a745d874fcea0fa9dfc4649aa14 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:49.510174 2385 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9a3448a425c041258991dc60dd64ae7d (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.514993 1849 catalog_manager.cc:5484] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 reported cstate change: config changed from index 6 to 7, VOTER 9a3448a425c041258991dc60dd64ae7d (127.31.161.198) evicted. New cstate: current_term: 1 leader_uuid: "269c9061e1a1435e92f79f0b9486dfea" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.515242 5525 raft_consensus.cc:2932] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9a3448a425c041258991dc60dd64ae7d (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.519191 2630 tablet_service.cc:1514] Processing DeleteTablet for tablet 74753a745d874fcea0fa9dfc4649aa14 with delete_type TABLET_DATA_TOMBSTONED (TS 9a3448a425c041258991dc60dd64ae7d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52494
I20240502 01:53:49.521867 5932 tablet_replica.cc:330] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d: stopping tablet replica
I20240502 01:53:49.522079 5932 raft_consensus.cc:2229] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:49.522377 5932 raft_consensus.cc:2258] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:49.524011 5932 ts_tablet_manager.cc:1875] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:49.528100 5932 ts_tablet_manager.cc:1888] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:53:49.528209 5932 log.cc:1196] T 74753a745d874fcea0fa9dfc4649aa14 P 9a3448a425c041258991dc60dd64ae7d: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal/wals/74753a745d874fcea0fa9dfc4649aa14
I20240502 01:53:49.528821 1834 catalog_manager.cc:4830] TS 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585): tablet 74753a745d874fcea0fa9dfc4649aa14 (table TestTable [id=21d6c8edffeb445cac75c180ed630ada]) successfully deleted
I20240502 01:53:49.645385 2385 consensus_queue.cc:237] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 283, Committed index: 283, Last appended: 1.283, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:49.648082 2516 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 283. Preceding OpId from leader: term: 1 index: 284. (index mismatch)
I20240502 01:53:49.648109 3604 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 283. Preceding OpId from leader: term: 1 index: 284. (index mismatch)
W20240502 01:53:49.648283 2322 consensus_peers.cc:485] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a -> Peer 9715934145414f2496a4105e95aa0987 (127.31.161.200:36271): Couldn't send request to peer 9715934145414f2496a4105e95aa0987. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0225b684df4f41499f76fe10b72a8861. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:49.648748 5710 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s
I20240502 01:53:49.648897 1985 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 283. Preceding OpId from leader: term: 1 index: 284. (index mismatch)
I20240502 01:53:49.649168 2117 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 283. Preceding OpId from leader: term: 1 index: 284. (index mismatch)
I20240502 01:53:49.649353 5710 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s
I20240502 01:53:49.649822 5846 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s
I20240502 01:53:49.650095 5887 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 284, Last known committed idx: 283, Time since last communication: 0.000s
I20240502 01:53:49.652915 5887 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Committing config change with OpId 1.284: config changed from index 122 to 284, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:49.653566 3604 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.284: config changed from index 122 to 284, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:49.659258 1985 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.284: config changed from index 122 to 284, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:49.660388 1845 catalog_manager.cc:5484] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 reported cstate change: config changed from index 122 to 284, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "7ef43ea75ecb46ee905b781c91aed95a" committed_config { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:49.671030 5887 raft_consensus.cc:1050] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a: attempting to promote NON_VOTER 9715934145414f2496a4105e95aa0987 to VOTER
I20240502 01:53:49.671983 5887 consensus_queue.cc:237] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 267, Committed index: 267, Last appended: 1.267, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:49.672518 2516 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.284: config changed from index 122 to 284, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:49.689960 2254 raft_consensus.cc:1261] T 677599c7914a4bafb83995cf3240679d P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 267. Preceding OpId from leader: term: 1 index: 268. (index mismatch)
I20240502 01:53:49.690721 5981 consensus_queue.cc:1035] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I20240502 01:53:49.692215 1985 raft_consensus.cc:1261] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 267. Preceding OpId from leader: term: 1 index: 268. (index mismatch)
I20240502 01:53:49.692589 2647 raft_consensus.cc:1261] T 677599c7914a4bafb83995cf3240679d P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 267. Preceding OpId from leader: term: 1 index: 268. (index mismatch)
I20240502 01:53:49.693095 5887 consensus_queue.cc:1035] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I20240502 01:53:49.693455 5982 consensus_queue.cc:1035] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I20240502 01:53:49.709408 5525 raft_consensus.cc:1261] T 677599c7914a4bafb83995cf3240679d P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 267. Preceding OpId from leader: term: 1 index: 268. (index mismatch)
I20240502 01:53:49.710559 5982 consensus_queue.cc:1035] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I20240502 01:53:49.710489 2121 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.284: config changed from index 122 to 284, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:49.714031 2117 raft_consensus.cc:1261] T 677599c7914a4bafb83995cf3240679d P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 267. Preceding OpId from leader: term: 1 index: 268. (index mismatch)
I20240502 01:53:49.716017 5974 consensus_queue.cc:1035] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 268, Last known committed idx: 267, Time since last communication: 0.000s
I20240502 01:53:49.717799 5974 raft_consensus.cc:2932] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Committing config change with OpId 1.268: config changed from index 267 to 268, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.720582 2254 raft_consensus.cc:2932] T 677599c7914a4bafb83995cf3240679d P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.268: config changed from index 267 to 268, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.723815 1985 raft_consensus.cc:2932] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.268: config changed from index 267 to 268, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.723843 2647 raft_consensus.cc:2932] T 677599c7914a4bafb83995cf3240679d P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.268: config changed from index 267 to 268, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.728260 1848 catalog_manager.cc:5484] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879 reported cstate change: config changed from index 267 to 268, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ef43ea75ecb46ee905b781c91aed95a" committed_config { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.735297 5525 raft_consensus.cc:2932] T 677599c7914a4bafb83995cf3240679d P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Committing config change with OpId 1.268: config changed from index 267 to 268, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.741420 2121 raft_consensus.cc:2932] T 677599c7914a4bafb83995cf3240679d P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.268: config changed from index 267 to 268, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 268 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.745712 2385 consensus_queue.cc:237] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 268, Committed index: 268, Last appended: 1.268, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:49.748291 5525 raft_consensus.cc:1261] T 677599c7914a4bafb83995cf3240679d P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 268. Preceding OpId from leader: term: 1 index: 269. (index mismatch)
I20240502 01:53:49.748425 2647 raft_consensus.cc:1261] T 677599c7914a4bafb83995cf3240679d P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 268. Preceding OpId from leader: term: 1 index: 269. (index mismatch)
I20240502 01:53:49.748425 2254 raft_consensus.cc:1261] T 677599c7914a4bafb83995cf3240679d P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 268. Preceding OpId from leader: term: 1 index: 269. (index mismatch)
I20240502 01:53:49.748696 2121 raft_consensus.cc:1261] T 677599c7914a4bafb83995cf3240679d P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 268. Preceding OpId from leader: term: 1 index: 269. (index mismatch)
I20240502 01:53:49.749235 5982 consensus_queue.cc:1035] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.000s
I20240502 01:53:49.749665 5981 consensus_queue.cc:1035] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.000s
I20240502 01:53:49.749949 5846 consensus_queue.cc:1035] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.000s
I20240502 01:53:49.750784 5982 consensus_queue.cc:1035] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.000s
I20240502 01:53:49.753571 5981 raft_consensus.cc:2932] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Committing config change with OpId 1.269: config changed from index 268 to 269, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.754341 5525 raft_consensus.cc:2932] T 677599c7914a4bafb83995cf3240679d P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Committing config change with OpId 1.269: config changed from index 268 to 269, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.754974 2647 raft_consensus.cc:2932] T 677599c7914a4bafb83995cf3240679d P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.269: config changed from index 268 to 269, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.758430 2254 raft_consensus.cc:2932] T 677599c7914a4bafb83995cf3240679d P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.269: config changed from index 268 to 269, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.761153 1841 catalog_manager.cc:5484] T 677599c7914a4bafb83995cf3240679d P 9a3448a425c041258991dc60dd64ae7d reported cstate change: config changed from index 268 to 269, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "7ef43ea75ecb46ee905b781c91aed95a" committed_config { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.764490 1835 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 677599c7914a4bafb83995cf3240679d with cas_config_opid_index 268: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:49.765540 2121 raft_consensus.cc:2932] T 677599c7914a4bafb83995cf3240679d P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.269: config changed from index 268 to 269, VOTER 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.772310 1971 tablet_service.cc:1514] Processing DeleteTablet for tablet 677599c7914a4bafb83995cf3240679d with delete_type TABLET_DATA_TOMBSTONED (TS 58dae5dc94444d5e92e6907c5b637879 not found in new config with opid_index 269) from {username='slave'} at 127.0.0.1:42016
I20240502 01:53:49.773532 5886 raft_consensus.cc:1050] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119: attempting to promote NON_VOTER 9715934145414f2496a4105e95aa0987 to VOTER
I20240502 01:53:49.774619 5886 consensus_queue.cc:237] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 346, Committed index: 346, Last appended: 1.346, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:49.781622 5999 tablet_replica.cc:330] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879: stopping tablet replica
I20240502 01:53:49.781865 5999 raft_consensus.cc:2229] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:49.782099 5999 raft_consensus.cc:2258] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:49.783650 5999 ts_tablet_manager.cc:1875] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:49.785439 2254 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 346. Preceding OpId from leader: term: 1 index: 347. (index mismatch)
I20240502 01:53:49.786856 2121 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 346. Preceding OpId from leader: term: 1 index: 347. (index mismatch)
I20240502 01:53:49.787549 5996 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
I20240502 01:53:49.788086 5525 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 346. Preceding OpId from leader: term: 1 index: 347. (index mismatch)
I20240502 01:53:49.789119 5995 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
I20240502 01:53:49.789464 6004 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
I20240502 01:53:49.794236 1985 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 346. Preceding OpId from leader: term: 1 index: 347. (index mismatch)
I20240502 01:53:49.794862 2384 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 346. Preceding OpId from leader: term: 1 index: 347. (index mismatch)
I20240502 01:53:49.795511 5999 ts_tablet_manager.cc:1888] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.268
I20240502 01:53:49.795504 5995 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
I20240502 01:53:49.795660 5999 log.cc:1196] T 677599c7914a4bafb83995cf3240679d P 58dae5dc94444d5e92e6907c5b637879: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/677599c7914a4bafb83995cf3240679d
I20240502 01:53:49.796360 5886 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 347, Last known committed idx: 346, Time since last communication: 0.000s
I20240502 01:53:49.797571 1836 catalog_manager.cc:4830] TS 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): tablet 677599c7914a4bafb83995cf3240679d (table rebalance_test_table_2 [id=832ef7a5c81541c3a3c9b450b6af8fc1]) successfully deleted
I20240502 01:53:49.817449 6003 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [term 1 LEADER]: Committing config change with OpId 1.347: config changed from index 346 to 347, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.818069 1985 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.347: config changed from index 346 to 347, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.819052 2384 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.347: config changed from index 346 to 347, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.821385 5525 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Committing config change with OpId 1.347: config changed from index 346 to 347, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.821449 2121 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.347: config changed from index 346 to 347, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.822584 1849 catalog_manager.cc:5484] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 reported cstate change: config changed from index 346 to 347, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d99e1fdda704211b3d101f892c57119" committed_config { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:49.832819 2254 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.347: config changed from index 346 to 347, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 347 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.836896 2516 consensus_queue.cc:237] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 347, Committed index: 347, Last appended: 1.347, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:49.839217 5525 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 347. Preceding OpId from leader: term: 1 index: 348. (index mismatch)
I20240502 01:53:49.839749 2254 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 347. Preceding OpId from leader: term: 1 index: 348. (index mismatch)
I20240502 01:53:49.840183 5996 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 348, Last known committed idx: 347, Time since last communication: 0.000s
I20240502 01:53:49.840584 1985 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 347. Preceding OpId from leader: term: 1 index: 348. (index mismatch)
I20240502 01:53:49.840569 6004 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 348, Last known committed idx: 347, Time since last communication: 0.000s
I20240502 01:53:49.840646 2384 raft_consensus.cc:1261] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 347. Preceding OpId from leader: term: 1 index: 348. (index mismatch)
I20240502 01:53:49.841128 6003 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 348, Last known committed idx: 347, Time since last communication: 0.000s
I20240502 01:53:49.841321 6004 consensus_queue.cc:1035] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 348, Last known committed idx: 347, Time since last communication: 0.000s
I20240502 01:53:49.843983 5995 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 [term 1 LEADER]: Committing config change with OpId 1.348: config changed from index 347 to 348, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.844254 1985 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.348: config changed from index 347 to 348, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.844664 5525 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Committing config change with OpId 1.348: config changed from index 347 to 348, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.848052 1834 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e4672bc55a874f159bf10855f0698ba1 with cas_config_opid_index 347: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:49.849153 2384 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.348: config changed from index 347 to 348, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.850088 1848 catalog_manager.cc:5484] T e4672bc55a874f159bf10855f0698ba1 P 9715934145414f2496a4105e95aa0987 reported cstate change: config changed from index 347 to 348, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New cstate: current_term: 1 leader_uuid: "4d99e1fdda704211b3d101f892c57119" committed_config { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.850447 2254 raft_consensus.cc:2932] T e4672bc55a874f159bf10855f0698ba1 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.348: config changed from index 347 to 348, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New config: { opid_index: 348 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.855338 2103 tablet_service.cc:1514] Processing DeleteTablet for tablet e4672bc55a874f159bf10855f0698ba1 with delete_type TABLET_DATA_TOMBSTONED (TS 269c9061e1a1435e92f79f0b9486dfea not found in new config with opid_index 348) from {username='slave'} at 127.0.0.1:50686
I20240502 01:53:49.857556 6015 tablet_replica.cc:330] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea: stopping tablet replica
I20240502 01:53:49.857738 6015 raft_consensus.cc:2229] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:49.858009 6015 raft_consensus.cc:2258] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:49.859512 6015 ts_tablet_manager.cc:1875] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:49.864358 6015 ts_tablet_manager.cc:1888] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.347
I20240502 01:53:49.864462 6015 log.cc:1196] T e4672bc55a874f159bf10855f0698ba1 P 269c9061e1a1435e92f79f0b9486dfea: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/wals/e4672bc55a874f159bf10855f0698ba1
I20240502 01:53:49.865265 1837 catalog_manager.cc:4830] TS 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621): tablet e4672bc55a874f159bf10855f0698ba1 (table rebalance_test_table_3 [id=3402138c5f114264ac2eb0f198dbe6b3]) successfully deleted
I20240502 01:53:49.903038 6017 raft_consensus.cc:1050] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d: attempting to promote NON_VOTER 9715934145414f2496a4105e95aa0987 to VOTER
I20240502 01:53:49.903856 6017 consensus_queue.cc:237] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 302, Committed index: 302, Last appended: 1.302, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:49.906673 2121 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 1 index: 303. (index mismatch)
I20240502 01:53:49.907860 3597 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 1 index: 303. (index mismatch)
I20240502 01:53:49.907945 2384 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 1 index: 303. (index mismatch)
I20240502 01:53:49.908363 5525 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 1 index: 303. (index mismatch)
I20240502 01:53:49.907389 2254 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 302. Preceding OpId from leader: term: 1 index: 303. (index mismatch)
I20240502 01:53:49.909001 6016 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.002s
I20240502 01:53:49.909317 6017 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20240502 01:53:49.909591 6019 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20240502 01:53:49.909845 6020 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20240502 01:53:49.910181 5914 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 303, Last known committed idx: 302, Time since last communication: 0.000s
I20240502 01:53:49.919246 5914 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [term 1 LEADER]: Committing config change with OpId 1.303: config changed from index 302 to 303, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.919948 5525 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Committing config change with OpId 1.303: config changed from index 302 to 303, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.924729 2253 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.303: config changed from index 302 to 303, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.926820 1848 catalog_manager.cc:5484] T e567c877188549fc965637e89bf437ae P 9715934145414f2496a4105e95aa0987 reported cstate change: config changed from index 302 to 303, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9a3448a425c041258991dc60dd64ae7d" committed_config { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.934562 2384 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.303: config changed from index 302 to 303, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.934805 2121 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.303: config changed from index 302 to 303, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.937425 3597 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.303: config changed from index 302 to 303, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 303 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.939246 6027 raft_consensus.cc:1050] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906: attempting to promote NON_VOTER 9715934145414f2496a4105e95aa0987 to VOTER
I20240502 01:53:49.940016 6027 consensus_queue.cc:237] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 328, Committed index: 328, Last appended: 1.328, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:49.945065 1985 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 328. Preceding OpId from leader: term: 1 index: 329. (index mismatch)
I20240502 01:53:49.945751 2384 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 328. Preceding OpId from leader: term: 1 index: 329. (index mismatch)
I20240502 01:53:49.945767 5524 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 328. Preceding OpId from leader: term: 1 index: 329. (index mismatch)
I20240502 01:53:49.946485 5917 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 329, Last known committed idx: 328, Time since last communication: 0.000s
I20240502 01:53:49.946722 3597 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 328. Preceding OpId from leader: term: 1 index: 329. (index mismatch)
I20240502 01:53:49.947047 5917 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 329, Last known committed idx: 328, Time since last communication: 0.000s
I20240502 01:53:49.947337 6026 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 329, Last known committed idx: 328, Time since last communication: 0.000s
I20240502 01:53:49.949507 2647 consensus_queue.cc:237] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 303, Committed index: 303, Last appended: 1.303, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:49.951551 3597 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 303. Preceding OpId from leader: term: 1 index: 304. (index mismatch)
I20240502 01:53:49.951633 2121 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 303. Preceding OpId from leader: term: 1 index: 304. (index mismatch)
I20240502 01:53:49.952051 2384 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 303. Preceding OpId from leader: term: 1 index: 304. (index mismatch)
I20240502 01:53:49.952500 6017 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.000s
I20240502 01:53:49.952659 5525 raft_consensus.cc:1261] T e567c877188549fc965637e89bf437ae P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Refusing update from remote peer 9a3448a425c041258991dc60dd64ae7d: Log matching property violated. Preceding OpId in replica: term: 1 index: 303. Preceding OpId from leader: term: 1 index: 304. (index mismatch)
I20240502 01:53:49.952843 6020 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.000s
I20240502 01:53:49.953157 5914 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.000s
I20240502 01:53:49.953531 6019 consensus_queue.cc:1035] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 304, Last known committed idx: 303, Time since last communication: 0.000s
I20240502 01:53:49.959039 6019 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d [term 1 LEADER]: Committing config change with OpId 1.304: config changed from index 303 to 304, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.959779 3597 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.304: config changed from index 303 to 304, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.961198 2384 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.304: config changed from index 303 to 304, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.964834 1834 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e567c877188549fc965637e89bf437ae with cas_config_opid_index 303: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:49.964737 2121 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.304: config changed from index 303 to 304, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.965415 1849 catalog_manager.cc:5484] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d reported cstate change: config changed from index 303 to 304, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New cstate: current_term: 1 leader_uuid: "9a3448a425c041258991dc60dd64ae7d" committed_config { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:49.971176 2234 tablet_service.cc:1514] Processing DeleteTablet for tablet e567c877188549fc965637e89bf437ae with delete_type TABLET_DATA_TOMBSTONED (TS 446c7f9bf31a4acc9dae8daf43330906 not found in new config with opid_index 304) from {username='slave'} at 127.0.0.1:45438
I20240502 01:53:49.976263 6032 tablet_replica.cc:330] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906: stopping tablet replica
I20240502 01:53:49.976528 6032 raft_consensus.cc:2229] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:49.976879 6032 raft_consensus.cc:2258] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:49.976707 5525 raft_consensus.cc:2932] T e567c877188549fc965637e89bf437ae P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Committing config change with OpId 1.304: config changed from index 303 to 304, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New config: { opid_index: 304 OBSOLETE_local: false peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.979259 6032 ts_tablet_manager.cc:1875] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:49.979463 2516 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 328. Preceding OpId from leader: term: 1 index: 329. (index mismatch)
I20240502 01:53:49.980122 5917 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 329, Last known committed idx: 328, Time since last communication: 0.000s
I20240502 01:53:49.985707 6032 ts_tablet_manager.cc:1888] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.303
I20240502 01:53:49.985846 6032 log.cc:1196] T e567c877188549fc965637e89bf437ae P 446c7f9bf31a4acc9dae8daf43330906: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/wals/e567c877188549fc965637e89bf437ae
I20240502 01:53:49.986584 1834 catalog_manager.cc:4830] TS 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353): tablet e567c877188549fc965637e89bf437ae (table rebalance_test_table_4 [id=2cd052d4ae9b43129f79f513560af30d]) successfully deleted
I20240502 01:53:49.988798 6027 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 329, Last known committed idx: 328, Time since last communication: 0.000s
I20240502 01:53:49.988699 6029 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEADER]: Committing config change with OpId 1.329: config changed from index 328 to 329, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.991256 3597 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.329: config changed from index 328 to 329, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.992668 2384 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.329: config changed from index 328 to 329, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.993762 2516 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.329: config changed from index 328 to 329, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:49.994356 1849 catalog_manager.cc:5484] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 reported cstate change: config changed from index 328 to 329, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "446c7f9bf31a4acc9dae8daf43330906" committed_config { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:50.000739 5524 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Committing config change with OpId 1.329: config changed from index 328 to 329, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:50.004349 1985 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.329: config changed from index 328 to 329, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 329 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:50.100708 1985 consensus_queue.cc:237] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 372, Committed index: 372, Last appended: 1.372, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false replace: true } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:50.104712 3597 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 372. Preceding OpId from leader: term: 1 index: 373. (index mismatch)
I20240502 01:53:50.106379 5378 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 373, Last known committed idx: 372, Time since last communication: 0.000s
I20240502 01:53:50.118206 6072 ts_tablet_manager.cc:902] T 0225b684df4f41499f76fe10b72a8861 P 9715934145414f2496a4105e95aa0987: Initiating tablet copy from peer 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225)
I20240502 01:53:50.125744 2384 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 372. Preceding OpId from leader: term: 1 index: 373. (index mismatch)
I20240502 01:53:50.125757 2647 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 372. Preceding OpId from leader: term: 1 index: 373. (index mismatch)
I20240502 01:53:50.126358 6068 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 373, Last known committed idx: 372, Time since last communication: 0.000s
I20240502 01:53:50.126902 2253 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 372. Preceding OpId from leader: term: 1 index: 373. (index mismatch)
I20240502 01:53:50.127957 5378 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 373, Last known committed idx: 372, Time since last communication: 0.000s
I20240502 01:53:50.128412 5378 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 373, Last known committed idx: 372, Time since last communication: 0.000s
I20240502 01:53:50.128871 6072 tablet_copy_client.cc:308] T 0225b684df4f41499f76fe10b72a8861 P 9715934145414f2496a4105e95aa0987: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:39225
I20240502 01:53:50.129554 2408 tablet_copy_service.cc:140] P 7ef43ea75ecb46ee905b781c91aed95a: Received BeginTabletCopySession request for tablet 0225b684df4f41499f76fe10b72a8861 from peer 9715934145414f2496a4105e95aa0987 ({username='slave'} at 127.31.161.200:50229)
I20240502 01:53:50.129703 2408 tablet_copy_service.cc:161] P 7ef43ea75ecb46ee905b781c91aed95a: Beginning new tablet copy session on tablet 0225b684df4f41499f76fe10b72a8861 from peer 9715934145414f2496a4105e95aa0987 at {username='slave'} at 127.31.161.200:50229: session id = 9715934145414f2496a4105e95aa0987-0225b684df4f41499f76fe10b72a8861
I20240502 01:53:50.131819 2408 tablet_copy_source_session.cc:205] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:50.133050 5378 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [term 1 LEADER]: Committing config change with OpId 1.373: config changed from index 239 to 373, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false replace: true } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:50.137342 2647 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.373: config changed from index 239 to 373, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false replace: true } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:50.139853 6072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0225b684df4f41499f76fe10b72a8861. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.147455 1848 catalog_manager.cc:5484] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d reported cstate change: config changed from index 239 to 373, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "58dae5dc94444d5e92e6907c5b637879" committed_config { opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false replace: true } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:50.150369 6072 tablet_copy_client.cc:783] T 0225b684df4f41499f76fe10b72a8861 P 9715934145414f2496a4105e95aa0987: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:50.150701 6072 tablet_copy_client.cc:647] T 0225b684df4f41499f76fe10b72a8861 P 9715934145414f2496a4105e95aa0987: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:50.154778 6072 tablet_copy_client.cc:523] T 0225b684df4f41499f76fe10b72a8861 P 9715934145414f2496a4105e95aa0987: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:50.154892 2253 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.373: config changed from index 239 to 373, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false replace: true } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
W20240502 01:53:50.156548 1928 consensus_peers.cc:485] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 -> Peer 9715934145414f2496a4105e95aa0987 (127.31.161.200:36271): Couldn't send request to peer 9715934145414f2496a4105e95aa0987. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4abedcf755774255867d46154f8131c7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:50.159327 6072 tablet_bootstrap.cc:492] T 0225b684df4f41499f76fe10b72a8861 P 9715934145414f2496a4105e95aa0987: Bootstrap starting.
I20240502 01:53:50.167119 3597 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.373: config changed from index 239 to 373, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false replace: true } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:50.180892 2384 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.373: config changed from index 239 to 373, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false replace: true } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:50.208384 2384 consensus_queue.cc:237] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 348, Committed index: 348, Last appended: 1.348, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:50.210999 2121 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 348. Preceding OpId from leader: term: 1 index: 349. (index mismatch)
W20240502 01:53:50.211087 2322 consensus_peers.cc:485] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a -> Peer 9715934145414f2496a4105e95aa0987 (127.31.161.200:36271): Couldn't send request to peer 9715934145414f2496a4105e95aa0987. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4220d1a63bd74ad0a5cac5a2f1bb988c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:50.211242 1985 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 348. Preceding OpId from leader: term: 1 index: 349. (index mismatch)
I20240502 01:53:50.211833 5974 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 349, Last known committed idx: 348, Time since last communication: 0.000s
I20240502 01:53:50.211982 3597 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 348. Preceding OpId from leader: term: 1 index: 349. (index mismatch)
I20240502 01:53:50.212348 5982 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 349, Last known committed idx: 348, Time since last communication: 0.000s
I20240502 01:53:50.212698 5974 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 349, Last known committed idx: 348, Time since last communication: 0.000s
I20240502 01:53:50.213430 2647 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 348. Preceding OpId from leader: term: 1 index: 349. (index mismatch)
I20240502 01:53:50.214224 5982 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 349, Last known committed idx: 348, Time since last communication: 0.000s
I20240502 01:53:50.218115 5982 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Committing config change with OpId 1.349: config changed from index 115 to 349, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:50.221627 2647 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.349: config changed from index 115 to 349, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:50.224341 1985 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.349: config changed from index 115 to 349, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:50.225533 3597 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.349: config changed from index 115 to 349, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:50.226154 1848 catalog_manager.cc:5484] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a reported cstate change: config changed from index 115 to 349, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "7ef43ea75ecb46ee905b781c91aed95a" committed_config { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:50.239130 2121 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.349: config changed from index 115 to 349, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:50.255998 2516 consensus_queue.cc:237] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 353, Committed index: 353, Last appended: 1.353, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
W20240502 01:53:50.259230 2454 consensus_peers.cc:485] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 -> Peer 9715934145414f2496a4105e95aa0987 (127.31.161.200:36271): Couldn't send request to peer 9715934145414f2496a4105e95aa0987. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73f01fd0dc7f488ca215b7f95ea8d988. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:50.259524 1985 raft_consensus.cc:1261] T 73f01fd0dc7f488ca215b7f95ea8d988 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 353. Preceding OpId from leader: term: 1 index: 354. (index mismatch)
I20240502 01:53:50.259524 2647 raft_consensus.cc:1261] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 353. Preceding OpId from leader: term: 1 index: 354. (index mismatch)
I20240502 01:53:50.260174 2253 raft_consensus.cc:1261] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 353. Preceding OpId from leader: term: 1 index: 354. (index mismatch)
I20240502 01:53:50.261065 5995 consensus_queue.cc:1035] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I20240502 01:53:50.261417 2121 raft_consensus.cc:1261] T 73f01fd0dc7f488ca215b7f95ea8d988 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 353. Preceding OpId from leader: term: 1 index: 354. (index mismatch)
I20240502 01:53:50.262110 5996 consensus_queue.cc:1035] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I20240502 01:53:50.262367 6003 consensus_queue.cc:1035] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I20240502 01:53:50.262614 5995 consensus_queue.cc:1035] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I20240502 01:53:50.277254 6003 raft_consensus.cc:2932] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [term 1 LEADER]: Committing config change with OpId 1.354: config changed from index -1 to 354, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:50.283565 2647 raft_consensus.cc:2932] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.354: config changed from index -1 to 354, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:50.284132 2254 raft_consensus.cc:2932] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.354: config changed from index -1 to 354, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:50.288228 1848 catalog_manager.cc:5484] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9a3448a425c041258991dc60dd64ae7d reported cstate change: config changed from index -1 to 354, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "4d99e1fdda704211b3d101f892c57119" committed_config { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:50.290409 2121 raft_consensus.cc:2932] T 73f01fd0dc7f488ca215b7f95ea8d988 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.354: config changed from index -1 to 354, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:50.292407 1985 raft_consensus.cc:2932] T 73f01fd0dc7f488ca215b7f95ea8d988 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.354: config changed from index -1 to 354, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:50.480091 6072 tablet_bootstrap.cc:492] T 0225b684df4f41499f76fe10b72a8861 P 9715934145414f2496a4105e95aa0987: Bootstrap replayed 1/1 log segments. Stats: ops{read=284 overwritten=0 applied=284 ignored=0} inserts{seen=966 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:50.480566 6072 tablet_bootstrap.cc:492] T 0225b684df4f41499f76fe10b72a8861 P 9715934145414f2496a4105e95aa0987: Bootstrap complete.
I20240502 01:53:50.481024 6072 ts_tablet_manager.cc:1367] T 0225b684df4f41499f76fe10b72a8861 P 9715934145414f2496a4105e95aa0987: Time spent bootstrapping tablet: real 0.322s user 0.286s sys 0.021s
I20240502 01:53:50.481858 6072 raft_consensus.cc:348] T 0225b684df4f41499f76fe10b72a8861 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:50.482148 6072 raft_consensus.cc:729] T 0225b684df4f41499f76fe10b72a8861 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9715934145414f2496a4105e95aa0987, State: Initialized, Role: LEARNER
I20240502 01:53:50.482365 6072 consensus_queue.cc:260] T 0225b684df4f41499f76fe10b72a8861 P 9715934145414f2496a4105e95aa0987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 284, Last appended: 1.284, Last appended by leader: 284, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 284 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:50.482975 6072 ts_tablet_manager.cc:1398] T 0225b684df4f41499f76fe10b72a8861 P 9715934145414f2496a4105e95aa0987: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:53:50.483855 2408 tablet_copy_service.cc:342] P 7ef43ea75ecb46ee905b781c91aed95a: Request end of tablet copy session 9715934145414f2496a4105e95aa0987-0225b684df4f41499f76fe10b72a8861 received from {username='slave'} at 127.31.161.200:50229
I20240502 01:53:50.483995 2408 tablet_copy_service.cc:433] P 7ef43ea75ecb46ee905b781c91aed95a: ending tablet copy session 9715934145414f2496a4105e95aa0987-0225b684df4f41499f76fe10b72a8861 on tablet 0225b684df4f41499f76fe10b72a8861 with peer 9715934145414f2496a4105e95aa0987
I20240502 01:53:50.524549 5524 raft_consensus.cc:1203] T 0225b684df4f41499f76fe10b72a8861 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.283->[1.284-1.284] Dedup: 1.284->[]
I20240502 01:53:50.617889 2253 consensus_queue.cc:237] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 329, Committed index: 329, Last appended: 1.329, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:50.619949 3597 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 329. Preceding OpId from leader: term: 1 index: 330. (index mismatch)
I20240502 01:53:50.620035 5524 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 329. Preceding OpId from leader: term: 1 index: 330. (index mismatch)
I20240502 01:53:50.620386 1985 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 329. Preceding OpId from leader: term: 1 index: 330. (index mismatch)
I20240502 01:53:50.620611 2384 raft_consensus.cc:1261] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 329. Preceding OpId from leader: term: 1 index: 330. (index mismatch)
I20240502 01:53:50.620647 6132 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 330, Last known committed idx: 329, Time since last communication: 0.000s
I20240502 01:53:50.620956 6027 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 330, Last known committed idx: 329, Time since last communication: 0.000s
I20240502 01:53:50.621301 6131 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 330, Last known committed idx: 329, Time since last communication: 0.000s
I20240502 01:53:50.621531 6132 consensus_queue.cc:1035] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 330, Last known committed idx: 329, Time since last communication: 0.000s
I20240502 01:53:50.624367 6132 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEADER]: Committing config change with OpId 1.330: config changed from index 329 to 330, VOTER 4d99e1fdda704211b3d101f892c57119 (127.31.161.197) evicted. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:50.626379 3597 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.330: config changed from index 329 to 330, VOTER 4d99e1fdda704211b3d101f892c57119 (127.31.161.197) evicted. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:50.628542 1985 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.330: config changed from index 329 to 330, VOTER 4d99e1fdda704211b3d101f892c57119 (127.31.161.197) evicted. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:50.630329 5524 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Committing config change with OpId 1.330: config changed from index 329 to 330, VOTER 4d99e1fdda704211b3d101f892c57119 (127.31.161.197) evicted. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:50.633985 1849 catalog_manager.cc:5484] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 reported cstate change: config changed from index 329 to 330, VOTER 4d99e1fdda704211b3d101f892c57119 (127.31.161.197) evicted. New cstate: current_term: 1 leader_uuid: "446c7f9bf31a4acc9dae8daf43330906" committed_config { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:50.634718 1834 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d592edf3298942cd93739aa846f867a4 with cas_config_opid_index 329: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:50.635960 2384 raft_consensus.cc:2932] T d592edf3298942cd93739aa846f867a4 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.330: config changed from index 329 to 330, VOTER 4d99e1fdda704211b3d101f892c57119 (127.31.161.197) evicted. New config: { opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:50.640805 2496 tablet_service.cc:1514] Processing DeleteTablet for tablet d592edf3298942cd93739aa846f867a4 with delete_type TABLET_DATA_TOMBSTONED (TS 4d99e1fdda704211b3d101f892c57119 not found in new config with opid_index 330) from {username='slave'} at 127.0.0.1:48022
I20240502 01:53:50.641610 6160 tablet_replica.cc:330] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119: stopping tablet replica
I20240502 01:53:50.641844 6160 raft_consensus.cc:2229] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:50.642213 6160 raft_consensus.cc:2258] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:50.643555 6160 ts_tablet_manager.cc:1875] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:50.645030 6072 ts_tablet_manager.cc:902] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987: Initiating tablet copy from peer 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307)
I20240502 01:53:50.649446 6072 tablet_copy_client.cc:308] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.193:39307
I20240502 01:53:50.649855 6160 ts_tablet_manager.cc:1888] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.329
I20240502 01:53:50.650070 6160 log.cc:1196] T d592edf3298942cd93739aa846f867a4 P 4d99e1fdda704211b3d101f892c57119: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal/wals/d592edf3298942cd93739aa846f867a4
I20240502 01:53:50.651181 1834 catalog_manager.cc:4830] TS 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987): tablet d592edf3298942cd93739aa846f867a4 (table rebalance_test_table_1 [id=db056d9436d84c2e8d32bb9d34b3c3a1]) successfully deleted
I20240502 01:53:50.656132 2012 tablet_copy_service.cc:140] P 58dae5dc94444d5e92e6907c5b637879: Received BeginTabletCopySession request for tablet 4abedcf755774255867d46154f8131c7 from peer 9715934145414f2496a4105e95aa0987 ({username='slave'} at 127.31.161.200:52999)
I20240502 01:53:50.656275 2012 tablet_copy_service.cc:161] P 58dae5dc94444d5e92e6907c5b637879: Beginning new tablet copy session on tablet 4abedcf755774255867d46154f8131c7 from peer 9715934145414f2496a4105e95aa0987 at {username='slave'} at 127.31.161.200:52999: session id = 9715934145414f2496a4105e95aa0987-4abedcf755774255867d46154f8131c7
I20240502 01:53:50.658565 2012 tablet_copy_source_session.cc:205] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:50.659703 6072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4abedcf755774255867d46154f8131c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.663884 6072 tablet_copy_client.cc:783] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:50.664095 6072 tablet_copy_client.cc:647] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:50.666630 6072 tablet_copy_client.cc:523] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:50.669389 6072 tablet_bootstrap.cc:492] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987: Bootstrap starting.
I20240502 01:53:50.721237 6166 ts_tablet_manager.cc:902] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9715934145414f2496a4105e95aa0987: Initiating tablet copy from peer 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987)
I20240502 01:53:50.723016 6166 tablet_copy_client.cc:308] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9715934145414f2496a4105e95aa0987: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.197:38987
I20240502 01:53:50.723783 2539 tablet_copy_service.cc:140] P 4d99e1fdda704211b3d101f892c57119: Received BeginTabletCopySession request for tablet 73f01fd0dc7f488ca215b7f95ea8d988 from peer 9715934145414f2496a4105e95aa0987 ({username='slave'} at 127.31.161.200:50693)
I20240502 01:53:50.723943 2539 tablet_copy_service.cc:161] P 4d99e1fdda704211b3d101f892c57119: Beginning new tablet copy session on tablet 73f01fd0dc7f488ca215b7f95ea8d988 from peer 9715934145414f2496a4105e95aa0987 at {username='slave'} at 127.31.161.200:50693: session id = 9715934145414f2496a4105e95aa0987-73f01fd0dc7f488ca215b7f95ea8d988
I20240502 01:53:50.725984 2539 tablet_copy_source_session.cc:205] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:50.727193 6166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73f01fd0dc7f488ca215b7f95ea8d988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.732501 6166 tablet_copy_client.cc:783] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9715934145414f2496a4105e95aa0987: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:50.732774 6166 tablet_copy_client.cc:647] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9715934145414f2496a4105e95aa0987: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:50.735455 6166 tablet_copy_client.cc:523] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9715934145414f2496a4105e95aa0987: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:50.738065 6166 tablet_bootstrap.cc:492] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9715934145414f2496a4105e95aa0987: Bootstrap starting.
I20240502 01:53:50.803140 6175 ts_tablet_manager.cc:902] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987: Initiating tablet copy from peer 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225)
I20240502 01:53:50.818560 6175 tablet_copy_client.cc:308] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:39225
I20240502 01:53:50.819388 2408 tablet_copy_service.cc:140] P 7ef43ea75ecb46ee905b781c91aed95a: Received BeginTabletCopySession request for tablet 4220d1a63bd74ad0a5cac5a2f1bb988c from peer 9715934145414f2496a4105e95aa0987 ({username='slave'} at 127.31.161.200:50229)
I20240502 01:53:50.819572 2408 tablet_copy_service.cc:161] P 7ef43ea75ecb46ee905b781c91aed95a: Beginning new tablet copy session on tablet 4220d1a63bd74ad0a5cac5a2f1bb988c from peer 9715934145414f2496a4105e95aa0987 at {username='slave'} at 127.31.161.200:50229: session id = 9715934145414f2496a4105e95aa0987-4220d1a63bd74ad0a5cac5a2f1bb988c
I20240502 01:53:50.822305 2408 tablet_copy_source_session.cc:205] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:50.823876 6175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4220d1a63bd74ad0a5cac5a2f1bb988c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:50.831245 6175 tablet_copy_client.cc:783] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:50.831528 6175 tablet_copy_client.cc:647] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:50.836766 6175 tablet_copy_client.cc:523] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:50.842969 6175 tablet_bootstrap.cc:492] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987: Bootstrap starting.
I20240502 01:53:51.042244 5982 raft_consensus.cc:1050] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a: attempting to promote NON_VOTER 9715934145414f2496a4105e95aa0987 to VOTER
I20240502 01:53:51.043335 5982 consensus_queue.cc:237] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 284, Committed index: 284, Last appended: 1.284, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:51.054514 5524 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 284. Preceding OpId from leader: term: 1 index: 285. (index mismatch)
I20240502 01:53:51.055295 5710 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 285, Last known committed idx: 284, Time since last communication: 0.000s
I20240502 01:53:51.062273 2516 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 284. Preceding OpId from leader: term: 1 index: 285. (index mismatch)
I20240502 01:53:51.063457 5710 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 285, Last known committed idx: 284, Time since last communication: 0.000s
I20240502 01:53:51.084354 3597 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 284. Preceding OpId from leader: term: 1 index: 285. (index mismatch)
I20240502 01:53:51.096144 6207 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 285, Last known committed idx: 284, Time since last communication: 0.000s
I20240502 01:53:51.111003 2121 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 284. Preceding OpId from leader: term: 1 index: 285. (index mismatch)
I20240502 01:53:51.112082 6211 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 285, Last known committed idx: 284, Time since last communication: 0.000s
I20240502 01:53:51.119230 1989 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 284. Preceding OpId from leader: term: 1 index: 285. (index mismatch)
I20240502 01:53:51.122151 6207 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 285, Last known committed idx: 284, Time since last communication: 0.000s
I20240502 01:53:51.132601 6072 tablet_bootstrap.cc:492] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987: Bootstrap replayed 1/1 log segments. Stats: ops{read=373 overwritten=0 applied=373 ignored=0} inserts{seen=1001 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:51.133244 6072 tablet_bootstrap.cc:492] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987: Bootstrap complete.
I20240502 01:53:51.133622 6072 ts_tablet_manager.cc:1367] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987: Time spent bootstrapping tablet: real 0.464s user 0.393s sys 0.040s
I20240502 01:53:51.134717 6072 raft_consensus.cc:348] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false replace: true } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:51.135272 6072 raft_consensus.cc:729] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9715934145414f2496a4105e95aa0987, State: Initialized, Role: LEARNER
I20240502 01:53:51.135597 6072 consensus_queue.cc:260] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 373, Last appended: 1.373, Last appended by leader: 373, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 373 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false replace: true } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:51.170028 6072 ts_tablet_manager.cc:1398] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987: Time spent starting tablet: real 0.036s user 0.022s sys 0.011s
I20240502 01:53:51.170854 2012 tablet_copy_service.cc:342] P 58dae5dc94444d5e92e6907c5b637879: Request end of tablet copy session 9715934145414f2496a4105e95aa0987-4abedcf755774255867d46154f8131c7 received from {username='slave'} at 127.31.161.200:52999
I20240502 01:53:51.171003 2012 tablet_copy_service.cc:433] P 58dae5dc94444d5e92e6907c5b637879: ending tablet copy session 9715934145414f2496a4105e95aa0987-4abedcf755774255867d46154f8131c7 on tablet 4abedcf755774255867d46154f8131c7 with peer 9715934145414f2496a4105e95aa0987
I20240502 01:53:51.181131 6207 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Committing config change with OpId 1.285: config changed from index 284 to 285, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.186282 5524 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Committing config change with OpId 1.285: config changed from index 284 to 285, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.186556 3597 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.285: config changed from index 284 to 285, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.192519 1848 catalog_manager.cc:5484] T 0225b684df4f41499f76fe10b72a8861 P 9715934145414f2496a4105e95aa0987 reported cstate change: config changed from index 284 to 285, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ef43ea75ecb46ee905b781c91aed95a" committed_config { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.194819 2117 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.285: config changed from index 284 to 285, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.205387 5524 raft_consensus.cc:1203] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.372->[1.373-1.373] Dedup: 1.373->[]
I20240502 01:53:51.191727 2516 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.285: config changed from index 284 to 285, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.222096 1989 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.285: config changed from index 284 to 285, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.255532 6166 tablet_bootstrap.cc:492] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9715934145414f2496a4105e95aa0987: Bootstrap replayed 1/1 log segments. Stats: ops{read=354 overwritten=0 applied=354 ignored=0} inserts{seen=1026 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:51.256054 6166 tablet_bootstrap.cc:492] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9715934145414f2496a4105e95aa0987: Bootstrap complete.
I20240502 01:53:51.256404 6166 ts_tablet_manager.cc:1367] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9715934145414f2496a4105e95aa0987: Time spent bootstrapping tablet: real 0.518s user 0.388s sys 0.030s
I20240502 01:53:51.257202 6166 raft_consensus.cc:348] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:51.257499 6166 raft_consensus.cc:729] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9715934145414f2496a4105e95aa0987, State: Initialized, Role: LEARNER
I20240502 01:53:51.257730 6166 consensus_queue.cc:260] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9715934145414f2496a4105e95aa0987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 354, Last appended: 1.354, Last appended by leader: 354, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:51.258416 6166 ts_tablet_manager.cc:1398] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9715934145414f2496a4105e95aa0987: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 01:53:51.259238 2539 tablet_copy_service.cc:342] P 4d99e1fdda704211b3d101f892c57119: Request end of tablet copy session 9715934145414f2496a4105e95aa0987-73f01fd0dc7f488ca215b7f95ea8d988 received from {username='slave'} at 127.31.161.200:50693
I20240502 01:53:51.259372 2539 tablet_copy_service.cc:433] P 4d99e1fdda704211b3d101f892c57119: ending tablet copy session 9715934145414f2496a4105e95aa0987-73f01fd0dc7f488ca215b7f95ea8d988 on tablet 73f01fd0dc7f488ca215b7f95ea8d988 with peer 9715934145414f2496a4105e95aa0987
I20240502 01:53:51.270063 2253 consensus_queue.cc:237] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 331, Committed index: 331, Last appended: 1.331, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
W20240502 01:53:51.273227 2190 consensus_peers.cc:485] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 -> Peer 9715934145414f2496a4105e95aa0987 (127.31.161.200:36271): Couldn't send request to peer 9715934145414f2496a4105e95aa0987. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4a07f0fe1964f958ad7ca4a63d054a6. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:51.273587 2647 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 331. Preceding OpId from leader: term: 1 index: 332. (index mismatch)
I20240502 01:53:51.273695 2516 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 331. Preceding OpId from leader: term: 1 index: 332. (index mismatch)
I20240502 01:53:51.274327 6159 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 332, Last known committed idx: 331, Time since last communication: 0.000s
I20240502 01:53:51.275017 1989 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 331. Preceding OpId from leader: term: 1 index: 332. (index mismatch)
I20240502 01:53:51.277225 6159 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 332, Last known committed idx: 331, Time since last communication: 0.000s
I20240502 01:53:51.277910 6159 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 332, Last known committed idx: 331, Time since last communication: 0.000s
I20240502 01:53:51.280140 3597 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 331. Preceding OpId from leader: term: 1 index: 332. (index mismatch)
I20240502 01:53:51.280768 6233 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 332, Last known committed idx: 331, Time since last communication: 0.000s
I20240502 01:53:51.281538 6159 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEADER]: Committing config change with OpId 1.332: config changed from index 106 to 332, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:51.282171 1989 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.332: config changed from index 106 to 332, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:51.287796 1844 catalog_manager.cc:5484] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879 reported cstate change: config changed from index 106 to 332, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "446c7f9bf31a4acc9dae8daf43330906" committed_config { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:51.291929 2516 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.332: config changed from index 106 to 332, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:51.302917 5524 raft_consensus.cc:1203] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.353->[1.354-1.354] Dedup: 1.354->[]
I20240502 01:53:51.305263 2647 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.332: config changed from index 106 to 332, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:51.305922 3597 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.332: config changed from index 106 to 332, NON_VOTER 9715934145414f2496a4105e95aa0987 (127.31.161.200) added. New config: { opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } } }
I20240502 01:53:51.372689 2384 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861"
dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39332
I20240502 01:53:51.372924 2384 raft_consensus.cc:595] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Received request to transfer leadership
I20240502 01:53:51.427137 6175 tablet_bootstrap.cc:492] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987: Bootstrap replayed 1/1 log segments. Stats: ops{read=349 overwritten=0 applied=349 ignored=0} inserts{seen=1031 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:51.427529 6175 tablet_bootstrap.cc:492] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987: Bootstrap complete.
I20240502 01:53:51.427798 6175 ts_tablet_manager.cc:1367] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987: Time spent bootstrapping tablet: real 0.585s user 0.373s sys 0.060s
I20240502 01:53:51.428406 6175 raft_consensus.cc:348] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:51.428627 6175 raft_consensus.cc:729] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9715934145414f2496a4105e95aa0987, State: Initialized, Role: LEARNER
I20240502 01:53:51.428810 6175 consensus_queue.cc:260] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 349, Last appended: 1.349, Last appended by leader: 349, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 349 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:51.429392 6175 ts_tablet_manager.cc:1398] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 01:53:51.430125 2408 tablet_copy_service.cc:342] P 7ef43ea75ecb46ee905b781c91aed95a: Request end of tablet copy session 9715934145414f2496a4105e95aa0987-4220d1a63bd74ad0a5cac5a2f1bb988c received from {username='slave'} at 127.31.161.200:50229
I20240502 01:53:51.430285 2408 tablet_copy_service.cc:433] P 7ef43ea75ecb46ee905b781c91aed95a: ending tablet copy session 9715934145414f2496a4105e95aa0987-4220d1a63bd74ad0a5cac5a2f1bb988c on tablet 4220d1a63bd74ad0a5cac5a2f1bb988c with peer 9715934145414f2496a4105e95aa0987
I20240502 01:53:51.589042 6208 raft_consensus.cc:979] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a: : Instructing follower 163649b5fbb140f18dccdf75f608b0f9 to start an election
I20240502 01:53:51.589251 6208 raft_consensus.cc:1067] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Signalling peer 163649b5fbb140f18dccdf75f608b0f9 to start an election
I20240502 01:53:51.590214 3597 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861"
dest_uuid: "163649b5fbb140f18dccdf75f608b0f9"
from {username='slave'} at 127.31.161.196:39531
I20240502 01:53:51.590570 3597 raft_consensus.cc:482] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:53:51.590754 3597 raft_consensus.cc:3037] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:51.593770 3597 raft_consensus.cc:504] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:51.596225 2384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861" candidate_uuid: "163649b5fbb140f18dccdf75f608b0f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 285 } } ignore_live_leader: true dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a"
I20240502 01:53:51.596328 2121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861" candidate_uuid: "163649b5fbb140f18dccdf75f608b0f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 285 } } ignore_live_leader: true dest_uuid: "269c9061e1a1435e92f79f0b9486dfea"
I20240502 01:53:51.596451 2384 raft_consensus.cc:3032] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:53:51.596392 1989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861" candidate_uuid: "163649b5fbb140f18dccdf75f608b0f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 285 } } ignore_live_leader: true dest_uuid: "58dae5dc94444d5e92e6907c5b637879"
I20240502 01:53:51.596513 2121 raft_consensus.cc:3037] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:51.596531 2384 raft_consensus.cc:729] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7ef43ea75ecb46ee905b781c91aed95a, State: Running, Role: LEADER
I20240502 01:53:51.596611 1989 raft_consensus.cc:3037] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:51.596856 2384 consensus_queue.cc:260] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 285, Committed index: 285, Last appended: 1.285, Last appended by leader: 285, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:51.597491 2384 raft_consensus.cc:3037] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:51.599354 1989 raft_consensus.cc:2454] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 163649b5fbb140f18dccdf75f608b0f9 in term 2.
I20240502 01:53:51.599396 2121 raft_consensus.cc:2454] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 163649b5fbb140f18dccdf75f608b0f9 in term 2.
I20240502 01:53:51.600054 2384 raft_consensus.cc:2454] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 163649b5fbb140f18dccdf75f608b0f9 in term 2.
I20240502 01:53:51.600464 3597 leader_election.cc:288] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [CANDIDATE]: Term 2 election: Requested vote from peers 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987), 9715934145414f2496a4105e95aa0987 (127.31.161.200:36271)
I20240502 01:53:51.600602 3538 leader_election.cc:302] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 6 voters: 4 yes votes; 0 no votes. yes voters: 163649b5fbb140f18dccdf75f608b0f9, 269c9061e1a1435e92f79f0b9486dfea, 58dae5dc94444d5e92e6907c5b637879, 7ef43ea75ecb46ee905b781c91aed95a; no voters:
I20240502 01:53:51.600867 2516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861" candidate_uuid: "163649b5fbb140f18dccdf75f608b0f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 285 } } ignore_live_leader: true dest_uuid: "4d99e1fdda704211b3d101f892c57119"
I20240502 01:53:51.601014 6222 raft_consensus.cc:2781] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:53:51.601101 2516 raft_consensus.cc:3037] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:51.603922 2516 raft_consensus.cc:2454] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 163649b5fbb140f18dccdf75f608b0f9 in term 2.
I20240502 01:53:51.613572 6222 raft_consensus.cc:686] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [term 2 LEADER]: Becoming Leader. State: Replica: 163649b5fbb140f18dccdf75f608b0f9, State: Running, Role: LEADER
I20240502 01:53:51.614468 6222 consensus_queue.cc:237] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 285, Committed index: 285, Last appended: 1.285, Last appended by leader: 285, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:51.615090 5524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0225b684df4f41499f76fe10b72a8861" candidate_uuid: "163649b5fbb140f18dccdf75f608b0f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 285 } } ignore_live_leader: true dest_uuid: "9715934145414f2496a4105e95aa0987"
I20240502 01:53:51.615377 5524 raft_consensus.cc:3037] T 0225b684df4f41499f76fe10b72a8861 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:53:51.617686 5524 raft_consensus.cc:2454] T 0225b684df4f41499f76fe10b72a8861 P 9715934145414f2496a4105e95aa0987 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 163649b5fbb140f18dccdf75f608b0f9 in term 2.
I20240502 01:53:51.621953 1844 catalog_manager.cc:5484] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 reported cstate change: term changed from 1 to 2, leader changed from 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) to 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199). New cstate: current_term: 2 leader_uuid: "163649b5fbb140f18dccdf75f608b0f9" committed_config { opid_index: 285 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:53:51.794587 6220 raft_consensus.cc:1050] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879: attempting to promote NON_VOTER 9715934145414f2496a4105e95aa0987 to VOTER
I20240502 01:53:51.795415 6220 consensus_queue.cc:237] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 373, Committed index: 373, Last appended: 1.373, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false replace: true } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:51.797793 5524 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 373. Preceding OpId from leader: term: 1 index: 374. (index mismatch)
I20240502 01:53:51.798115 2384 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 373. Preceding OpId from leader: term: 1 index: 374. (index mismatch)
I20240502 01:53:51.798200 3597 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 373. Preceding OpId from leader: term: 1 index: 374. (index mismatch)
I20240502 01:53:51.798441 6302 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 374, Last known committed idx: 373, Time since last communication: 0.000s
I20240502 01:53:51.799119 6301 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 374, Last known committed idx: 373, Time since last communication: 0.000s
I20240502 01:53:51.799927 2647 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 373. Preceding OpId from leader: term: 1 index: 374. (index mismatch)
I20240502 01:53:51.800944 2253 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 373. Preceding OpId from leader: term: 1 index: 374. (index mismatch)
I20240502 01:53:51.805647 6302 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 374, Last known committed idx: 373, Time since last communication: 0.000s
I20240502 01:53:51.813143 6305 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 374, Last known committed idx: 373, Time since last communication: 0.000s
I20240502 01:53:51.815419 6299 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 374, Last known committed idx: 373, Time since last communication: 0.000s
I20240502 01:53:51.831100 6302 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [term 1 LEADER]: Committing config change with OpId 1.374: config changed from index 373 to 374, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false replace: true } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.832432 3597 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.374: config changed from index 373 to 374, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false replace: true } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.833359 2647 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.374: config changed from index 373 to 374, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false replace: true } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.833416 2384 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.374: config changed from index 373 to 374, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false replace: true } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.835935 5524 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Committing config change with OpId 1.374: config changed from index 373 to 374, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false replace: true } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.839504 1844 catalog_manager.cc:5484] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a reported cstate change: config changed from index 373 to 374, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58dae5dc94444d5e92e6907c5b637879" committed_config { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false replace: true } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.841432 2253 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.374: config changed from index 373 to 374, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 374 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false replace: true } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.852068 5524 raft_consensus.cc:1203] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.348->[1.349-1.349] Dedup: 1.349->[]
I20240502 01:53:51.859242 6175 ts_tablet_manager.cc:902] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987: Initiating tablet copy from peer 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353)
I20240502 01:53:51.866601 6175 tablet_copy_client.cc:308] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:39353
I20240502 01:53:51.867280 2275 tablet_copy_service.cc:140] P 446c7f9bf31a4acc9dae8daf43330906: Received BeginTabletCopySession request for tablet f4a07f0fe1964f958ad7ca4a63d054a6 from peer 9715934145414f2496a4105e95aa0987 ({username='slave'} at 127.31.161.200:43913)
I20240502 01:53:51.867440 2275 tablet_copy_service.cc:161] P 446c7f9bf31a4acc9dae8daf43330906: Beginning new tablet copy session on tablet f4a07f0fe1964f958ad7ca4a63d054a6 from peer 9715934145414f2496a4105e95aa0987 at {username='slave'} at 127.31.161.200:43913: session id = 9715934145414f2496a4105e95aa0987-f4a07f0fe1964f958ad7ca4a63d054a6
I20240502 01:53:51.870383 2275 tablet_copy_source_session.cc:205] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:53:51.870476 6307 raft_consensus.cc:1050] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119: attempting to promote NON_VOTER 9715934145414f2496a4105e95aa0987 to VOTER
I20240502 01:53:51.871423 6307 consensus_queue.cc:237] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 354, Committed index: 354, Last appended: 1.354, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:51.871803 6175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4a07f0fe1964f958ad7ca4a63d054a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:53:51.874531 5524 raft_consensus.cc:1261] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 354. Preceding OpId from leader: term: 1 index: 355. (index mismatch)
I20240502 01:53:51.874702 1989 raft_consensus.cc:1261] T 73f01fd0dc7f488ca215b7f95ea8d988 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 354. Preceding OpId from leader: term: 1 index: 355. (index mismatch)
I20240502 01:53:51.875247 6003 consensus_queue.cc:1035] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 355, Last known committed idx: 354, Time since last communication: 0.000s
I20240502 01:53:51.878154 6175 tablet_copy_client.cc:783] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987: tablet copy: Starting download of 0 data blocks...
I20240502 01:53:51.878398 6175 tablet_copy_client.cc:647] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987: tablet copy: Starting download of 1 WAL segments...
I20240502 01:53:51.878852 6003 consensus_queue.cc:1035] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 355, Last known committed idx: 354, Time since last communication: 0.000s
I20240502 01:53:51.880532 2647 raft_consensus.cc:1261] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 354. Preceding OpId from leader: term: 1 index: 355. (index mismatch)
I20240502 01:53:51.880607 2253 raft_consensus.cc:1261] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 354. Preceding OpId from leader: term: 1 index: 355. (index mismatch)
I20240502 01:53:51.881196 6003 consensus_queue.cc:1035] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 355, Last known committed idx: 354, Time since last communication: 0.000s
I20240502 01:53:51.881640 6003 consensus_queue.cc:1035] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 355, Last known committed idx: 354, Time since last communication: 0.000s
I20240502 01:53:51.885666 6175 tablet_copy_client.cc:523] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:53:51.887352 2121 raft_consensus.cc:1261] T 73f01fd0dc7f488ca215b7f95ea8d988 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 354. Preceding OpId from leader: term: 1 index: 355. (index mismatch)
I20240502 01:53:51.888046 6307 consensus_queue.cc:1035] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 355, Last known committed idx: 354, Time since last communication: 0.000s
I20240502 01:53:51.890048 6175 tablet_bootstrap.cc:492] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987: Bootstrap starting.
I20240502 01:53:51.914976 6324 raft_consensus.cc:2932] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [term 1 LEADER]: Committing config change with OpId 1.355: config changed from index 354 to 355, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.915462 2647 raft_consensus.cc:2932] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.355: config changed from index 354 to 355, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.918118 2253 raft_consensus.cc:2932] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.355: config changed from index 354 to 355, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.918970 2121 raft_consensus.cc:2932] T 73f01fd0dc7f488ca215b7f95ea8d988 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.355: config changed from index 354 to 355, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.920334 5524 raft_consensus.cc:2932] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Committing config change with OpId 1.355: config changed from index 354 to 355, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.926332 1845 catalog_manager.cc:5484] T 73f01fd0dc7f488ca215b7f95ea8d988 P 269c9061e1a1435e92f79f0b9486dfea reported cstate change: config changed from index 354 to 355, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d99e1fdda704211b3d101f892c57119" committed_config { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.927780 1989 raft_consensus.cc:2932] T 73f01fd0dc7f488ca215b7f95ea8d988 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.355: config changed from index 354 to 355, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 355 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.950666 2516 consensus_queue.cc:237] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 355, Committed index: 355, Last appended: 1.355, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:51.954213 2647 raft_consensus.cc:1261] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 355. Preceding OpId from leader: term: 1 index: 356. (index mismatch)
I20240502 01:53:51.954859 2121 raft_consensus.cc:1261] T 73f01fd0dc7f488ca215b7f95ea8d988 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 355. Preceding OpId from leader: term: 1 index: 356. (index mismatch)
I20240502 01:53:51.954887 6307 consensus_queue.cc:1035] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 356, Last known committed idx: 355, Time since last communication: 0.000s
I20240502 01:53:51.955478 6307 consensus_queue.cc:1035] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 356, Last known committed idx: 355, Time since last communication: 0.000s
I20240502 01:53:51.957149 5524 raft_consensus.cc:1261] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 355. Preceding OpId from leader: term: 1 index: 356. (index mismatch)
I20240502 01:53:51.958088 6324 consensus_queue.cc:1035] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 356, Last known committed idx: 355, Time since last communication: 0.000s
I20240502 01:53:51.959359 6307 raft_consensus.cc:2932] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [term 1 LEADER]: Committing config change with OpId 1.356: config changed from index 355 to 356, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.960283 1989 raft_consensus.cc:1261] T 73f01fd0dc7f488ca215b7f95ea8d988 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 4d99e1fdda704211b3d101f892c57119: Log matching property violated. Preceding OpId in replica: term: 1 index: 355. Preceding OpId from leader: term: 1 index: 356. (index mismatch)
I20240502 01:53:51.959898 2121 raft_consensus.cc:2932] T 73f01fd0dc7f488ca215b7f95ea8d988 P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.356: config changed from index 355 to 356, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.964610 5525 raft_consensus.cc:2932] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Committing config change with OpId 1.356: config changed from index 355 to 356, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.965564 1841 catalog_manager.cc:5484] T 73f01fd0dc7f488ca215b7f95ea8d988 P 269c9061e1a1435e92f79f0b9486dfea reported cstate change: config changed from index 355 to 356, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New cstate: current_term: 1 leader_uuid: "4d99e1fdda704211b3d101f892c57119" committed_config { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.969383 2647 raft_consensus.cc:2932] T 73f01fd0dc7f488ca215b7f95ea8d988 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.356: config changed from index 355 to 356, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.971040 6324 consensus_queue.cc:1035] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 356, Last known committed idx: 355, Time since last communication: 0.000s
I20240502 01:53:51.971138 2234 tablet_service.cc:1514] Processing DeleteTablet for tablet 73f01fd0dc7f488ca215b7f95ea8d988 with delete_type TABLET_DATA_TOMBSTONED (TS 446c7f9bf31a4acc9dae8daf43330906 not found in new config with opid_index 356) from {username='slave'} at 127.0.0.1:45438
I20240502 01:53:51.971812 6341 tablet_replica.cc:330] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906: stopping tablet replica
I20240502 01:53:51.972018 6341 raft_consensus.cc:2229] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:51.972285 6341 raft_consensus.cc:2258] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:51.972388 1834 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 73f01fd0dc7f488ca215b7f95ea8d988 with cas_config_opid_index 355: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:51.973960 1989 raft_consensus.cc:2932] T 73f01fd0dc7f488ca215b7f95ea8d988 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.356: config changed from index 355 to 356, VOTER 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195) evicted. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:51.974542 6341 ts_tablet_manager.cc:1875] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:51.991592 6341 ts_tablet_manager.cc:1888] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.355
I20240502 01:53:51.991720 6341 log.cc:1196] T 73f01fd0dc7f488ca215b7f95ea8d988 P 446c7f9bf31a4acc9dae8daf43330906: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/wals/73f01fd0dc7f488ca215b7f95ea8d988
I20240502 01:53:51.992569 1834 catalog_manager.cc:4830] TS 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353): tablet 73f01fd0dc7f488ca215b7f95ea8d988 (table rebalance_test_table_0 [id=9b1fbaeec6af4ae5a31888af58780907]) successfully deleted
I20240502 01:53:52.113750 1989 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879 [term 2 FOLLOWER]: Refusing update from remote peer 163649b5fbb140f18dccdf75f608b0f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 285. Preceding OpId from leader: term: 2 index: 286. (index mismatch)
I20240502 01:53:52.114496 6222 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 286, Last known committed idx: 285, Time since last communication: 0.000s
I20240502 01:53:52.199571 2121 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 2 FOLLOWER]: Refusing update from remote peer 163649b5fbb140f18dccdf75f608b0f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 285. Preceding OpId from leader: term: 2 index: 286. (index mismatch)
I20240502 01:53:52.200189 6222 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 286, Last known committed idx: 285, Time since last communication: 0.000s
I20240502 01:53:52.207078 2384 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 2 FOLLOWER]: Refusing update from remote peer 163649b5fbb140f18dccdf75f608b0f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 285. Preceding OpId from leader: term: 2 index: 286. (index mismatch)
I20240502 01:53:52.207670 6222 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 286, Last known committed idx: 285, Time since last communication: 0.000s
I20240502 01:53:52.229825 2516 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 2 FOLLOWER]: Refusing update from remote peer 163649b5fbb140f18dccdf75f608b0f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 285. Preceding OpId from leader: term: 2 index: 286. (index mismatch)
I20240502 01:53:52.230038 5524 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 9715934145414f2496a4105e95aa0987 [term 2 FOLLOWER]: Refusing update from remote peer 163649b5fbb140f18dccdf75f608b0f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 285. Preceding OpId from leader: term: 2 index: 286. (index mismatch)
I20240502 01:53:52.230607 6222 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 286, Last known committed idx: 285, Time since last communication: 0.000s
I20240502 01:53:52.235246 6208 raft_consensus.cc:1050] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a: attempting to promote NON_VOTER 9715934145414f2496a4105e95aa0987 to VOTER
I20240502 01:53:52.236397 6208 consensus_queue.cc:237] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 349, Committed index: 349, Last appended: 1.349, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:52.238421 6222 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 286, Last known committed idx: 285, Time since last communication: 0.000s
I20240502 01:53:52.246819 1985 consensus_queue.cc:237] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 374, Committed index: 374, Last appended: 1.374, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:52.249704 5525 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 374. Preceding OpId from leader: term: 1 index: 375. (index mismatch)
I20240502 01:53:52.250026 2647 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 374. Preceding OpId from leader: term: 1 index: 375. (index mismatch)
I20240502 01:53:52.250595 2253 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 374. Preceding OpId from leader: term: 1 index: 375. (index mismatch)
I20240502 01:53:52.250653 6303 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 375, Last known committed idx: 374, Time since last communication: 0.000s
I20240502 01:53:52.252000 2385 raft_consensus.cc:1261] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Refusing update from remote peer 58dae5dc94444d5e92e6907c5b637879: Log matching property violated. Preceding OpId in replica: term: 1 index: 374. Preceding OpId from leader: term: 1 index: 375. (index mismatch)
I20240502 01:53:52.252607 6302 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 375, Last known committed idx: 374, Time since last communication: 0.000s
I20240502 01:53:52.253049 6302 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 375, Last known committed idx: 374, Time since last communication: 0.000s
I20240502 01:53:52.253805 6303 consensus_queue.cc:1035] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 375, Last known committed idx: 374, Time since last communication: 0.000s
I20240502 01:53:52.256727 6302 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 [term 1 LEADER]: Committing config change with OpId 1.375: config changed from index 374 to 375, VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) evicted. New config: { opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.261059 2253 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 FOLLOWER]: Committing config change with OpId 1.375: config changed from index 374 to 375, VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) evicted. New config: { opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.265470 2647 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.375: config changed from index 374 to 375, VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) evicted. New config: { opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.273909 6175 tablet_bootstrap.cc:492] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987: Bootstrap replayed 1/1 log segments. Stats: ops{read=332 overwritten=0 applied=332 ignored=0} inserts{seen=1001 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:53:52.274358 6175 tablet_bootstrap.cc:492] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987: Bootstrap complete.
I20240502 01:53:52.273957 1845 catalog_manager.cc:5484] T 4abedcf755774255867d46154f8131c7 P 58dae5dc94444d5e92e6907c5b637879 reported cstate change: config changed from index 374 to 375, VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) evicted. New cstate: current_term: 1 leader_uuid: "58dae5dc94444d5e92e6907c5b637879" committed_config { opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:52.274657 6175 ts_tablet_manager.cc:1367] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987: Time spent bootstrapping tablet: real 0.385s user 0.322s sys 0.047s
I20240502 01:53:52.275547 6175 raft_consensus.cc:348] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:52.275820 6175 raft_consensus.cc:729] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9715934145414f2496a4105e95aa0987, State: Initialized, Role: LEARNER
I20240502 01:53:52.276069 6175 consensus_queue.cc:260] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 332, Last appended: 1.332, Last appended by leader: 332, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 332 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: true } }
I20240502 01:53:52.276764 6175 ts_tablet_manager.cc:1398] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:53:52.277738 2275 tablet_copy_service.cc:342] P 446c7f9bf31a4acc9dae8daf43330906: Request end of tablet copy session 9715934145414f2496a4105e95aa0987-f4a07f0fe1964f958ad7ca4a63d054a6 received from {username='slave'} at 127.31.161.200:43913
I20240502 01:53:52.277873 2275 tablet_copy_service.cc:433] P 446c7f9bf31a4acc9dae8daf43330906: ending tablet copy session 9715934145414f2496a4105e95aa0987-f4a07f0fe1964f958ad7ca4a63d054a6 on tablet f4a07f0fe1964f958ad7ca4a63d054a6 with peer 9715934145414f2496a4105e95aa0987
I20240502 01:53:52.280561 5525 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Committing config change with OpId 1.375: config changed from index 374 to 375, VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) evicted. New config: { opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.283187 1836 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4abedcf755774255867d46154f8131c7 with cas_config_opid_index 374: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:52.289356 2117 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 349. Preceding OpId from leader: term: 1 index: 350. (index mismatch)
I20240502 01:53:52.289662 3597 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 349. Preceding OpId from leader: term: 1 index: 350. (index mismatch)
I20240502 01:53:52.290275 6211 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20240502 01:53:52.290623 6208 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20240502 01:53:52.291097 1985 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 349. Preceding OpId from leader: term: 1 index: 350. (index mismatch)
I20240502 01:53:52.291635 6312 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20240502 01:53:52.307699 5523 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 349. Preceding OpId from leader: term: 1 index: 350. (index mismatch)
I20240502 01:53:52.308346 5525 raft_consensus.cc:1203] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.331->[1.332-1.332] Dedup: 1.332->[]
I20240502 01:53:52.309177 6211 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20240502 01:53:52.325138 2647 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 349. Preceding OpId from leader: term: 1 index: 350. (index mismatch)
I20240502 01:53:52.326555 2384 raft_consensus.cc:2932] T 4abedcf755774255867d46154f8131c7 P 7ef43ea75ecb46ee905b781c91aed95a [term 1 FOLLOWER]: Committing config change with OpId 1.375: config changed from index 374 to 375, VOTER 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199) evicted. New config: { opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.332540 6208 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 350, Last known committed idx: 349, Time since last communication: 0.000s
I20240502 01:53:52.335402 6208 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Committing config change with OpId 1.350: config changed from index 349 to 350, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.341586 1985 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.350: config changed from index 349 to 350, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.340869 3604 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.350: config changed from index 349 to 350, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.347172 5523 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Committing config change with OpId 1.350: config changed from index 349 to 350, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.347594 1849 catalog_manager.cc:5484] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879 reported cstate change: config changed from index 349 to 350, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7ef43ea75ecb46ee905b781c91aed95a" committed_config { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.349211 3585 tablet_service.cc:1514] Processing DeleteTablet for tablet 4abedcf755774255867d46154f8131c7 with delete_type TABLET_DATA_TOMBSTONED (TS 163649b5fbb140f18dccdf75f608b0f9 not found in new config with opid_index 375) from {username='slave'} at 127.0.0.1:38772
I20240502 01:53:52.350163 6380 tablet_replica.cc:330] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9: stopping tablet replica
I20240502 01:53:52.350741 6380 raft_consensus.cc:2229] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:52.352675 6380 raft_consensus.cc:2258] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.355700 2117 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Committing config change with OpId 1.350: config changed from index 349 to 350, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.357755 6380 ts_tablet_manager.cc:1875] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:52.362082 2647 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.350: config changed from index 349 to 350, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 350 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.368260 2384 consensus_queue.cc:237] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 350, Committed index: 350, Last appended: 1.350, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:52.370563 2647 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 350. Preceding OpId from leader: term: 1 index: 351. (index mismatch)
I20240502 01:53:52.371142 3604 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 350. Preceding OpId from leader: term: 1 index: 351. (index mismatch)
I20240502 01:53:52.371616 6211 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20240502 01:53:52.371851 6208 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20240502 01:53:52.373765 5524 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 350. Preceding OpId from leader: term: 1 index: 351. (index mismatch)
I20240502 01:53:52.373777 1985 raft_consensus.cc:1261] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 7ef43ea75ecb46ee905b781c91aed95a: Log matching property violated. Preceding OpId in replica: term: 1 index: 350. Preceding OpId from leader: term: 1 index: 351. (index mismatch)
I20240502 01:53:52.374390 6312 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20240502 01:53:52.375589 6312 consensus_queue.cc:1035] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 351, Last known committed idx: 350, Time since last communication: 0.000s
I20240502 01:53:52.376407 6380 ts_tablet_manager.cc:1888] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.374
I20240502 01:53:52.376709 6380 log.cc:1196] T 4abedcf755774255867d46154f8131c7 P 163649b5fbb140f18dccdf75f608b0f9: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal/wals/4abedcf755774255867d46154f8131c7
I20240502 01:53:52.376888 6312 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a [term 1 LEADER]: Committing config change with OpId 1.351: config changed from index 350 to 351, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.377923 1836 catalog_manager.cc:4830] TS 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381): tablet 4abedcf755774255867d46154f8131c7 (table rebalance_test_table_0 [id=9b1fbaeec6af4ae5a31888af58780907]) successfully deleted
I20240502 01:53:52.380103 1835 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4220d1a63bd74ad0a5cac5a2f1bb988c with cas_config_opid_index 350: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:52.380789 1985 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.351: config changed from index 350 to 351, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.380453 5524 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Committing config change with OpId 1.351: config changed from index 350 to 351, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.381734 3597 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.351: config changed from index 350 to 351, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.382182 1845 catalog_manager.cc:5484] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a reported cstate change: config changed from index 350 to 351, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New cstate: current_term: 1 leader_uuid: "7ef43ea75ecb46ee905b781c91aed95a" committed_config { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:52.386626 2103 tablet_service.cc:1514] Processing DeleteTablet for tablet 4220d1a63bd74ad0a5cac5a2f1bb988c with delete_type TABLET_DATA_TOMBSTONED (TS 269c9061e1a1435e92f79f0b9486dfea not found in new config with opid_index 351) from {username='slave'} at 127.0.0.1:50686
I20240502 01:53:52.387315 6390 tablet_replica.cc:330] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea: stopping tablet replica
I20240502 01:53:52.388502 6390 raft_consensus.cc:2229] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:52.388901 6390 raft_consensus.cc:2258] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.392285 2647 raft_consensus.cc:2932] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.351: config changed from index 350 to 351, VOTER 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194) evicted. New config: { opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.393129 6390 ts_tablet_manager.cc:1875] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:52.399335 6390 ts_tablet_manager.cc:1888] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.350
I20240502 01:53:52.399457 6390 log.cc:1196] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 269c9061e1a1435e92f79f0b9486dfea: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/wals/4220d1a63bd74ad0a5cac5a2f1bb988c
I20240502 01:53:52.400216 1837 catalog_manager.cc:4830] TS 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621): tablet 4220d1a63bd74ad0a5cac5a2f1bb988c (table rebalance_test_table_3 [id=3402138c5f114264ac2eb0f198dbe6b3]) successfully deleted
I20240502 01:53:52.762462 3597 consensus_queue.cc:237] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 286, Committed index: 286, Last appended: 2.286, Last appended by leader: 285, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:52.764618 5524 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 9715934145414f2496a4105e95aa0987 [term 2 FOLLOWER]: Refusing update from remote peer 163649b5fbb140f18dccdf75f608b0f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 286. Preceding OpId from leader: term: 2 index: 287. (index mismatch)
I20240502 01:53:52.764672 2516 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 2 FOLLOWER]: Refusing update from remote peer 163649b5fbb140f18dccdf75f608b0f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 286. Preceding OpId from leader: term: 2 index: 287. (index mismatch)
I20240502 01:53:52.764936 1985 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879 [term 2 FOLLOWER]: Refusing update from remote peer 163649b5fbb140f18dccdf75f608b0f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 286. Preceding OpId from leader: term: 2 index: 287. (index mismatch)
I20240502 01:53:52.765251 6362 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 287, Last known committed idx: 286, Time since last communication: 0.000s
I20240502 01:53:52.765722 2117 raft_consensus.cc:1261] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 2 FOLLOWER]: Refusing update from remote peer 163649b5fbb140f18dccdf75f608b0f9: Log matching property violated. Preceding OpId in replica: term: 2 index: 286. Preceding OpId from leader: term: 2 index: 287. (index mismatch)
I20240502 01:53:52.765661 6355 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 287, Last known committed idx: 286, Time since last communication: 0.000s
I20240502 01:53:52.766072 6222 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 287, Last known committed idx: 286, Time since last communication: 0.000s
I20240502 01:53:52.766382 6367 consensus_queue.cc:1035] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 287, Last known committed idx: 286, Time since last communication: 0.000s
I20240502 01:53:52.768915 6222 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 [term 2 LEADER]: Committing config change with OpId 2.287: config changed from index 285 to 287, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.770484 2516 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 4d99e1fdda704211b3d101f892c57119 [term 2 FOLLOWER]: Committing config change with OpId 2.287: config changed from index 285 to 287, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.772176 1836 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0225b684df4f41499f76fe10b72a8861 with cas_config_opid_index 285: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:52.773396 1985 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 58dae5dc94444d5e92e6907c5b637879 [term 2 FOLLOWER]: Committing config change with OpId 2.287: config changed from index 285 to 287, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.773486 5524 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 9715934145414f2496a4105e95aa0987 [term 2 FOLLOWER]: Committing config change with OpId 2.287: config changed from index 285 to 287, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.775414 1845 catalog_manager.cc:5484] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 reported cstate change: config changed from index 285 to 287, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New cstate: current_term: 2 leader_uuid: "163649b5fbb140f18dccdf75f608b0f9" committed_config { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:52.777725 2117 raft_consensus.cc:2932] T 0225b684df4f41499f76fe10b72a8861 P 269c9061e1a1435e92f79f0b9486dfea [term 2 FOLLOWER]: Committing config change with OpId 2.287: config changed from index 285 to 287, VOTER 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196) evicted. New config: { opid_index: 287 OBSOLETE_local: false peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.780310 2366 tablet_service.cc:1514] Processing DeleteTablet for tablet 0225b684df4f41499f76fe10b72a8861 with delete_type TABLET_DATA_TOMBSTONED (TS 7ef43ea75ecb46ee905b781c91aed95a not found in new config with opid_index 287) from {username='slave'} at 127.0.0.1:54148
I20240502 01:53:52.780956 6424 tablet_replica.cc:330] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a: stopping tablet replica
I20240502 01:53:52.781205 6424 raft_consensus.cc:2229] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:52.781430 6424 raft_consensus.cc:2258] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.783591 6424 ts_tablet_manager.cc:1875] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:52.787770 6425 raft_consensus.cc:1050] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906: attempting to promote NON_VOTER 9715934145414f2496a4105e95aa0987 to VOTER
I20240502 01:53:52.788393 6424 ts_tablet_manager.cc:1888] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.286
I20240502 01:53:52.788475 6424 log.cc:1196] T 0225b684df4f41499f76fe10b72a8861 P 7ef43ea75ecb46ee905b781c91aed95a: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/0225b684df4f41499f76fe10b72a8861
I20240502 01:53:52.788671 6425 consensus_queue.cc:237] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 332, Committed index: 332, Last appended: 1.332, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:52.789144 1835 catalog_manager.cc:4830] TS 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225): tablet 0225b684df4f41499f76fe10b72a8861 (table rebalance_test_table_2 [id=832ef7a5c81541c3a3c9b450b6af8fc1]) successfully deleted
I20240502 01:53:52.791155 2647 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 332. Preceding OpId from leader: term: 1 index: 333. (index mismatch)
I20240502 01:53:52.791684 6427 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 333, Last known committed idx: 332, Time since last communication: 0.000s
I20240502 01:53:52.792713 2516 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 332. Preceding OpId from leader: term: 1 index: 333. (index mismatch)
I20240502 01:53:52.793705 5524 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987 [term 1 LEARNER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 332. Preceding OpId from leader: term: 1 index: 333. (index mismatch)
I20240502 01:53:52.794047 6370 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 333, Last known committed idx: 332, Time since last communication: 0.000s
I20240502 01:53:52.794471 6427 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 333, Last known committed idx: 332, Time since last communication: 0.000s
I20240502 01:53:52.795787 3597 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 332. Preceding OpId from leader: term: 1 index: 333. (index mismatch)
I20240502 01:53:52.795955 1985 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 332. Preceding OpId from leader: term: 1 index: 333. (index mismatch)
I20240502 01:53:52.796517 6427 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 333, Last known committed idx: 332, Time since last communication: 0.000s
I20240502 01:53:52.796828 6428 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 333, Last known committed idx: 332, Time since last communication: 0.000s
I20240502 01:53:52.798524 6428 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEADER]: Committing config change with OpId 1.333: config changed from index 332 to 333, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.800320 2516 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Committing config change with OpId 1.333: config changed from index 332 to 333, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.803262 2647 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.333: config changed from index 332 to 333, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.803627 5524 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Committing config change with OpId 1.333: config changed from index 332 to 333, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.806603 1841 catalog_manager.cc:5484] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119 reported cstate change: config changed from index 332 to 333, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "446c7f9bf31a4acc9dae8daf43330906" committed_config { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.807668 1985 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.333: config changed from index 332 to 333, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.814442 3604 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.333: config changed from index 332 to 333, 9715934145414f2496a4105e95aa0987 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 333 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } attrs { replace: true } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.820487 2253 consensus_queue.cc:237] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 333, Committed index: 333, Last appended: 1.333, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:52.822757 5524 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 333. Preceding OpId from leader: term: 1 index: 334. (index mismatch)
I20240502 01:53:52.823257 1985 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 333. Preceding OpId from leader: term: 1 index: 334. (index mismatch)
I20240502 01:53:52.823259 6428 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 334, Last known committed idx: 333, Time since last communication: 0.000s
I20240502 01:53:52.823513 2647 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 333. Preceding OpId from leader: term: 1 index: 334. (index mismatch)
I20240502 01:53:52.823971 6428 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 334, Last known committed idx: 333, Time since last communication: 0.000s
I20240502 01:53:52.824273 3604 raft_consensus.cc:1261] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Refusing update from remote peer 446c7f9bf31a4acc9dae8daf43330906: Log matching property violated. Preceding OpId in replica: term: 1 index: 333. Preceding OpId from leader: term: 1 index: 334. (index mismatch)
I20240502 01:53:52.824258 6425 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 334, Last known committed idx: 333, Time since last communication: 0.000s
I20240502 01:53:52.824779 6427 consensus_queue.cc:1035] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [LEADER]: Connected to new peer: Peer: permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 334, Last known committed idx: 333, Time since last communication: 0.000s
I20240502 01:53:52.826570 6427 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 [term 1 LEADER]: Committing config change with OpId 1.334: config changed from index 333 to 334, VOTER 4d99e1fdda704211b3d101f892c57119 (127.31.161.197) evicted. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.827466 1989 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 58dae5dc94444d5e92e6907c5b637879 [term 1 FOLLOWER]: Committing config change with OpId 1.334: config changed from index 333 to 334, VOTER 4d99e1fdda704211b3d101f892c57119 (127.31.161.197) evicted. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.828337 3597 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Committing config change with OpId 1.334: config changed from index 333 to 334, VOTER 4d99e1fdda704211b3d101f892c57119 (127.31.161.197) evicted. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.830607 1834 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f4a07f0fe1964f958ad7ca4a63d054a6 with cas_config_opid_index 333: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:53:52.829986 2647 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Committing config change with OpId 1.334: config changed from index 333 to 334, VOTER 4d99e1fdda704211b3d101f892c57119 (127.31.161.197) evicted. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.832489 1849 catalog_manager.cc:5484] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 reported cstate change: config changed from index 333 to 334, VOTER 4d99e1fdda704211b3d101f892c57119 (127.31.161.197) evicted. New cstate: current_term: 1 leader_uuid: "446c7f9bf31a4acc9dae8daf43330906" committed_config { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:53:52.833987 5524 raft_consensus.cc:2932] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Committing config change with OpId 1.334: config changed from index 333 to 334, VOTER 4d99e1fdda704211b3d101f892c57119 (127.31.161.197) evicted. New config: { opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } } }
I20240502 01:53:52.837368 2496 tablet_service.cc:1514] Processing DeleteTablet for tablet f4a07f0fe1964f958ad7ca4a63d054a6 with delete_type TABLET_DATA_TOMBSTONED (TS 4d99e1fdda704211b3d101f892c57119 not found in new config with opid_index 334) from {username='slave'} at 127.0.0.1:48022
I20240502 01:53:52.843789 6435 tablet_replica.cc:330] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119: stopping tablet replica
I20240502 01:53:52.844034 6435 raft_consensus.cc:2229] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:53:52.844313 6435 raft_consensus.cc:2258] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:53:52.846401 6435 ts_tablet_manager.cc:1875] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:53:52.850536 6435 ts_tablet_manager.cc:1888] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.333
I20240502 01:53:52.850618 6435 log.cc:1196] T f4a07f0fe1964f958ad7ca4a63d054a6 P 4d99e1fdda704211b3d101f892c57119: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal/wals/f4a07f0fe1964f958ad7ca4a63d054a6
I20240502 01:53:52.851251 1834 catalog_manager.cc:4830] TS 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987): tablet f4a07f0fe1964f958ad7ca4a63d054a6 (table rebalance_test_table_4 [id=2cd052d4ae9b43129f79f513560af30d]) successfully deleted
I20240502 01:53:52.993047 3585 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:53.000715 2103 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:53.013332 2234 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:53.020278 2366 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:53.037737 2496 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:53.042804 1971 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:53.047631 2630 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:53.069799 5505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:53.273265 3585 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:53.291575 1971 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:53.292222 2234 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:53.293325 2103 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:53.308445 2366 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:53.312471 2496 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:53.313988 2630 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:53.372510 5505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:53.702252 3585 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:53.723738 2103 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:53.735234 2234 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:53:53.768368 1971 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:53:53.780833 2366 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:53:53.790367 5505 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:53:53.811596 2630 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 01:53:53.876480 2496 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
4689208abb42472a9ef49d5db3ba5cda | 127.31.161.254:45525 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.31.161.212:35723 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
163649b5fbb140f18dccdf75f608b0f9 | 127.31.161.199:46381 | HEALTHY | <none> | 1 | 0
269c9061e1a1435e92f79f0b9486dfea | 127.31.161.194:39621 | HEALTHY | <none> | 1 | 0
446c7f9bf31a4acc9dae8daf43330906 | 127.31.161.195:39353 | HEALTHY | <none> | 2 | 0
4d99e1fdda704211b3d101f892c57119 | 127.31.161.197:38987 | HEALTHY | <none> | 4 | 0
58dae5dc94444d5e92e6907c5b637879 | 127.31.161.193:39307 | HEALTHY | <none> | 1 | 0
7ef43ea75ecb46ee905b781c91aed95a | 127.31.161.196:39225 | HEALTHY | <none> | 4 | 0
9715934145414f2496a4105e95aa0987 | 127.31.161.200:36271 | HEALTHY | <none> | 0 | 0
9a3448a425c041258991dc60dd64ae7d | 127.31.161.198:37585 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.31.161.193 | experimental | 127.31.161.193:39307
local_ip_for_outbound_sockets | 127.31.161.194 | experimental | 127.31.161.194:39621
local_ip_for_outbound_sockets | 127.31.161.195 | experimental | 127.31.161.195:39353
local_ip_for_outbound_sockets | 127.31.161.196 | experimental | 127.31.161.196:39225
local_ip_for_outbound_sockets | 127.31.161.197 | experimental | 127.31.161.197:38987
local_ip_for_outbound_sockets | 127.31.161.198 | experimental | 127.31.161.198:37585
local_ip_for_outbound_sockets | 127.31.161.199 | experimental | 127.31.161.199:46381
local_ip_for_outbound_sockets | 127.31.161.200 | experimental | 127.31.161.200:36271
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.31.161.193:39307
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.31.161.194:39621
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.31.161.195:39353
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.31.161.196:39225
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.31.161.197:38987
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.31.161.198:37585
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.31.161.199:46381
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.31.161.200:36271
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.31.161.212:35723 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 80
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 01:53:53.922988 32391 log_verifier.cc:126] Checking tablet 0225b684df4f41499f76fe10b72a8861
I20240502 01:53:54.182590 32391 log_verifier.cc:177] Verified matching terms for 287 ops in tablet 0225b684df4f41499f76fe10b72a8861
I20240502 01:53:54.183388 32391 log_verifier.cc:126] Checking tablet 1983e0a53ee54b62afa16676bf986332
I20240502 01:53:54.515707 32391 log_verifier.cc:177] Verified matching terms for 350 ops in tablet 1983e0a53ee54b62afa16676bf986332
I20240502 01:53:54.516516 32391 log_verifier.cc:126] Checking tablet 27398ad1eb7f4bfab185f8affe1f790d
I20240502 01:53:54.794960 32391 log_verifier.cc:177] Verified matching terms for 322 ops in tablet 27398ad1eb7f4bfab185f8affe1f790d
I20240502 01:53:54.795725 32391 log_verifier.cc:126] Checking tablet 341e0200ea654dcfb91133135b458a16
I20240502 01:53:55.144316 32391 log_verifier.cc:177] Verified matching terms for 379 ops in tablet 341e0200ea654dcfb91133135b458a16
I20240502 01:53:55.145359 32391 log_verifier.cc:126] Checking tablet 4068710224254825bf42671418592ece
I20240502 01:53:55.373924 32391 log_verifier.cc:177] Verified matching terms for 267 ops in tablet 4068710224254825bf42671418592ece
I20240502 01:53:55.374543 32391 log_verifier.cc:126] Checking tablet 4220d1a63bd74ad0a5cac5a2f1bb988c
I20240502 01:53:55.702476 32391 log_verifier.cc:177] Verified matching terms for 351 ops in tablet 4220d1a63bd74ad0a5cac5a2f1bb988c
I20240502 01:53:55.703274 32391 log_verifier.cc:126] Checking tablet 4abedcf755774255867d46154f8131c7
I20240502 01:53:56.018028 32391 log_verifier.cc:177] Verified matching terms for 375 ops in tablet 4abedcf755774255867d46154f8131c7
I20240502 01:53:56.018918 32391 log_verifier.cc:126] Checking tablet 677599c7914a4bafb83995cf3240679d
I20240502 01:53:56.282123 32391 log_verifier.cc:177] Verified matching terms for 269 ops in tablet 677599c7914a4bafb83995cf3240679d
I20240502 01:53:56.282819 32391 log_verifier.cc:126] Checking tablet 73f01fd0dc7f488ca215b7f95ea8d988
I20240502 01:53:56.582348 32391 log_verifier.cc:177] Verified matching terms for 356 ops in tablet 73f01fd0dc7f488ca215b7f95ea8d988
I20240502 01:53:56.610002 32391 log_verifier.cc:126] Checking tablet 74753a745d874fcea0fa9dfc4649aa14
I20240502 01:53:56.644901 32391 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 74753a745d874fcea0fa9dfc4649aa14
I20240502 01:53:56.645004 32391 log_verifier.cc:126] Checking tablet 9895c93b27764c22b77087d74e0149c6
I20240502 01:53:56.937407 32391 log_verifier.cc:177] Verified matching terms for 337 ops in tablet 9895c93b27764c22b77087d74e0149c6
I20240502 01:53:56.938187 32391 log_verifier.cc:126] Checking tablet c291b2a5e9d140f881b2bd54f8c9ed19
I20240502 01:53:57.229202 32391 log_verifier.cc:177] Verified matching terms for 316 ops in tablet c291b2a5e9d140f881b2bd54f8c9ed19
I20240502 01:53:57.229943 32391 log_verifier.cc:126] Checking tablet d592edf3298942cd93739aa846f867a4
I20240502 01:53:57.510303 32391 log_verifier.cc:177] Verified matching terms for 330 ops in tablet d592edf3298942cd93739aa846f867a4
I20240502 01:53:57.511180 32391 log_verifier.cc:126] Checking tablet e4672bc55a874f159bf10855f0698ba1
I20240502 01:53:57.834334 32391 log_verifier.cc:177] Verified matching terms for 348 ops in tablet e4672bc55a874f159bf10855f0698ba1
I20240502 01:53:57.835162 32391 log_verifier.cc:126] Checking tablet e567c877188549fc965637e89bf437ae
I20240502 01:53:58.104583 32391 log_verifier.cc:177] Verified matching terms for 304 ops in tablet e567c877188549fc965637e89bf437ae
I20240502 01:53:58.105351 32391 log_verifier.cc:126] Checking tablet f4a07f0fe1964f958ad7ca4a63d054a6
I20240502 01:53:58.417027 32391 log_verifier.cc:177] Verified matching terms for 334 ops in tablet f4a07f0fe1964f958ad7ca4a63d054a6
I20240502 01:53:58.425036 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 1910
I20240502 01:53:58.436808 2032 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:58.722379 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 1910
W20240502 01:53:58.766541 5461 connection.cc:535] server connection from 127.31.161.193:35405 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:53:58.767263 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 2041
W20240502 01:53:58.767442 2453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.161.193:39307 (error 108)
W20240502 01:53:58.769462 2453 consensus_peers.cc:485] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 -> Peer 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): Couldn't send request to peer 58dae5dc94444d5e92e6907c5b637879. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.193:39307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:58.776844 2163 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:53:58.798067 2453 consensus_peers.cc:485] T 4068710224254825bf42671418592ece P 4d99e1fdda704211b3d101f892c57119 -> Peer 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): Couldn't send request to peer 58dae5dc94444d5e92e6907c5b637879. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.193:39307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:58.928071 3539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.161.193:39307: connect: Connection refused (error 111)
W20240502 01:53:58.929685 3539 consensus_peers.cc:485] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 -> Peer 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): Couldn't send request to peer 58dae5dc94444d5e92e6907c5b637879. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.193:39307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:58.979620 2189 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.161.193:39307: connect: Connection refused (error 111)
W20240502 01:53:58.980926 2189 consensus_peers.cc:485] T f4a07f0fe1964f958ad7ca4a63d054a6 P 446c7f9bf31a4acc9dae8daf43330906 -> Peer 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): Couldn't send request to peer 58dae5dc94444d5e92e6907c5b637879. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.193:39307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:59.018873 2321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.161.193:39307: connect: Connection refused (error 111)
W20240502 01:53:59.019313 2189 consensus_peers.cc:485] T d592edf3298942cd93739aa846f867a4 P 446c7f9bf31a4acc9dae8daf43330906 -> Peer 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): Couldn't send request to peer 58dae5dc94444d5e92e6907c5b637879. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.193:39307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:59.020228 2321 consensus_peers.cc:485] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a -> Peer 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): Couldn't send request to peer 58dae5dc94444d5e92e6907c5b637879. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.193:39307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:59.062059 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 2041
W20240502 01:53:59.063730 2453 consensus_peers.cc:485] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 -> Peer 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): Couldn't send request to peer 58dae5dc94444d5e92e6907c5b637879. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.193:39307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:59.105579 2586 connection.cc:535] client connection to 127.31.161.194:39621 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:53:59.105645 2322 connection.cc:535] client connection to 127.31.161.194:39621 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:53:59.105849 2586 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:53:59.106299 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 2172
W20240502 01:53:59.107008 2322 consensus_peers.cc:485] T 677599c7914a4bafb83995cf3240679d P 7ef43ea75ecb46ee905b781c91aed95a -> Peer 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621): Couldn't send request to peer 269c9061e1a1435e92f79f0b9486dfea. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.194:39621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:59.107120 2322 consensus_peers.cc:485] T 27398ad1eb7f4bfab185f8affe1f790d P 7ef43ea75ecb46ee905b781c91aed95a -> Peer 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621): Couldn't send request to peer 269c9061e1a1435e92f79f0b9486dfea. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.194:39621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:59.108628 2586 consensus_peers.cc:485] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d -> Peer 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621): Couldn't send request to peer 269c9061e1a1435e92f79f0b9486dfea. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.194:39621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:59.108731 2586 consensus_peers.cc:485] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d -> Peer 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621): Couldn't send request to peer 269c9061e1a1435e92f79f0b9486dfea. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.194:39621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:59.109354 2454 consensus_peers.cc:485] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 -> Peer 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621): Couldn't send request to peer 269c9061e1a1435e92f79f0b9486dfea. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.194:39621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:59.114055 2295 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:53:59.121681 2322 consensus_peers.cc:485] T c291b2a5e9d140f881b2bd54f8c9ed19 P 7ef43ea75ecb46ee905b781c91aed95a -> Peer 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621): Couldn't send request to peer 269c9061e1a1435e92f79f0b9486dfea. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.194:39621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:59.132751 2321 consensus_peers.cc:485] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 7ef43ea75ecb46ee905b781c91aed95a -> Peer 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): Couldn't send request to peer 58dae5dc94444d5e92e6907c5b637879. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.193:39307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:59.229148 2586 consensus_peers.cc:485] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d -> Peer 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621): Couldn't send request to peer 269c9061e1a1435e92f79f0b9486dfea. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.194:39621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:59.250829 3540 consensus_peers.cc:485] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 -> Peer 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621): Couldn't send request to peer 269c9061e1a1435e92f79f0b9486dfea. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.194:39621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:59.291872 2454 consensus_peers.cc:485] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 -> Peer 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621): Couldn't send request to peer 269c9061e1a1435e92f79f0b9486dfea. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.194:39621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:59.311960 2453 consensus_peers.cc:485] T 73f01fd0dc7f488ca215b7f95ea8d988 P 4d99e1fdda704211b3d101f892c57119 -> Peer 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): Couldn't send request to peer 58dae5dc94444d5e92e6907c5b637879. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.193:39307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:59.404373 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 2172
I20240502 01:53:59.448395 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 2304
I20240502 01:53:59.454941 2427 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:53:59.619351 2583 consensus_peers.cc:485] T 9895c93b27764c22b77087d74e0149c6 P 9a3448a425c041258991dc60dd64ae7d -> Peer 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353): Couldn't send request to peer 446c7f9bf31a4acc9dae8daf43330906. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.195:39353: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:59.689744 6588 raft_consensus.cc:482] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58dae5dc94444d5e92e6907c5b637879)
I20240502 01:53:59.689953 6588 raft_consensus.cc:504] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:59.691078 6588 leader_election.cc:288] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 9715934145414f2496a4105e95aa0987 (127.31.161.200:36271)
I20240502 01:53:59.692644 5523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abedcf755774255867d46154f8131c7" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 2 candidate_status { last_received { term: 1 index: 375 } } ignore_live_leader: false dest_uuid: "9715934145414f2496a4105e95aa0987" is_pre_election: true
W20240502 01:53:59.693042 2585 leader_election.cc:334] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): Network error: Client connection negotiation failed: client connection to 127.31.161.193:39307: connect: Connection refused (error 111)
W20240502 01:53:59.694658 2451 consensus_peers.cc:485] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 -> Peer 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353): Couldn't send request to peer 446c7f9bf31a4acc9dae8daf43330906. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.195:39353: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:59.694805 2583 leader_election.cc:334] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353): Network error: Client connection negotiation failed: client connection to 127.31.161.195:39353: connect: Connection refused (error 111)
I20240502 01:53:59.694905 2583 leader_election.cc:302] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 9a3448a425c041258991dc60dd64ae7d; no voters: 446c7f9bf31a4acc9dae8daf43330906, 58dae5dc94444d5e92e6907c5b637879, 9715934145414f2496a4105e95aa0987
I20240502 01:53:59.695189 6588 raft_consensus.cc:2726] T 4abedcf755774255867d46154f8131c7 P 9a3448a425c041258991dc60dd64ae7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:53:59.756584 2384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4abedcf755774255867d46154f8131c7" candidate_uuid: "9a3448a425c041258991dc60dd64ae7d" candidate_term: 2 candidate_status { last_received { term: 1 index: 375 } } ignore_live_leader: false dest_uuid: "7ef43ea75ecb46ee905b781c91aed95a" is_pre_election: true
I20240502 01:53:59.757639 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 2304
W20240502 01:53:59.802631 2584 connection.cc:535] server connection from 127.31.161.196:58977 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:53:59.802958 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 2436
W20240502 01:53:59.803340 2452 consensus_peers.cc:485] T 341e0200ea654dcfb91133135b458a16 P 4d99e1fdda704211b3d101f892c57119 -> Peer 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225): Couldn't send request to peer 7ef43ea75ecb46ee905b781c91aed95a. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.196:39225: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:53:59.804068 2452 consensus_peers.cc:485] T e4672bc55a874f159bf10855f0698ba1 P 4d99e1fdda704211b3d101f892c57119 -> Peer 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225): Couldn't send request to peer 7ef43ea75ecb46ee905b781c91aed95a. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.196:39225: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:59.808763 2558 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:53:59.832857 6593 raft_consensus.cc:482] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 269c9061e1a1435e92f79f0b9486dfea)
I20240502 01:53:59.833125 6593 raft_consensus.cc:504] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:59.834246 6593 leader_election.cc:288] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987), 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621), 9715934145414f2496a4105e95aa0987 (127.31.161.200:36271)
W20240502 01:53:59.835644 3538 leader_election.cc:334] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225): Network error: Client connection negotiation failed: client connection to 127.31.161.196:39225: connect: Connection refused (error 111)
I20240502 01:53:59.835646 5523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74753a745d874fcea0fa9dfc4649aa14" candidate_uuid: "163649b5fbb140f18dccdf75f608b0f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "9715934145414f2496a4105e95aa0987" is_pre_election: true
W20240502 01:53:59.836091 3540 leader_election.cc:334] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621): Network error: Client connection negotiation failed: client connection to 127.31.161.194:39621: connect: Connection refused (error 111)
I20240502 01:53:59.836385 3540 leader_election.cc:302] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 163649b5fbb140f18dccdf75f608b0f9; no voters: 269c9061e1a1435e92f79f0b9486dfea, 7ef43ea75ecb46ee905b781c91aed95a, 9715934145414f2496a4105e95aa0987
I20240502 01:53:59.836658 6593 raft_consensus.cc:2726] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:53:59.854135 2584 consensus_peers.cc:485] T e567c877188549fc965637e89bf437ae P 9a3448a425c041258991dc60dd64ae7d -> Peer 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225): Couldn't send request to peer 7ef43ea75ecb46ee905b781c91aed95a. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.196:39225: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:53:59.902340 6602 raft_consensus.cc:482] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 269c9061e1a1435e92f79f0b9486dfea)
I20240502 01:53:59.902601 6602 raft_consensus.cc:504] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "4d99e1fdda704211b3d101f892c57119" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 38987 } } peers { permanent_uuid: "269c9061e1a1435e92f79f0b9486dfea" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 39621 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:53:59.910967 6602 leader_election.cc:288] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987), 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621), 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381)
W20240502 01:53:59.914247 5461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.161.194:39621: connect: Connection refused (error 111)
W20240502 01:53:59.917910 5461 leader_election.cc:334] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 269c9061e1a1435e92f79f0b9486dfea (127.31.161.194:39621): Network error: Client connection negotiation failed: client connection to 127.31.161.194:39621: connect: Connection refused (error 111)
W20240502 01:53:59.918218 5459 leader_election.cc:334] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225): Network error: Client connection negotiation failed: client connection to 127.31.161.196:39225: connect: Connection refused (error 111)
I20240502 01:53:59.921636 3604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74753a745d874fcea0fa9dfc4649aa14" candidate_uuid: "9715934145414f2496a4105e95aa0987" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "163649b5fbb140f18dccdf75f608b0f9" is_pre_election: true
I20240502 01:53:59.921957 3604 raft_consensus.cc:2454] T 74753a745d874fcea0fa9dfc4649aa14 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9715934145414f2496a4105e95aa0987 in term 1.
W20240502 01:53:59.958685 2583 consensus_peers.cc:485] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d -> Peer 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353): Couldn't send request to peer 446c7f9bf31a4acc9dae8daf43330906. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.195:39353: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:00.095696 2515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74753a745d874fcea0fa9dfc4649aa14" candidate_uuid: "163649b5fbb140f18dccdf75f608b0f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "4d99e1fdda704211b3d101f892c57119" is_pre_election: true
I20240502 01:54:00.097489 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 2436
W20240502 01:54:00.142572 2583 connection.cc:535] client connection to 127.31.161.197:38987 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:00.142591 1836 connection.cc:535] server connection from 127.31.161.197:45195 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:00.142733 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 2568
W20240502 01:54:00.143355 5459 connection.cc:535] server connection from 127.31.161.197:60899 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:00.142590 2584 connection.cc:535] server connection from 127.31.161.197:35711 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:00.143879 5458 leader_election.cc:334] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987): Network error: Client connection negotiation failed: client connection to 127.31.161.197:38987: connect: Connection refused (error 111)
W20240502 01:54:00.144083 2583 consensus_peers.cc:485] T 1983e0a53ee54b62afa16676bf986332 P 9a3448a425c041258991dc60dd64ae7d -> Peer 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987): Couldn't send request to peer 4d99e1fdda704211b3d101f892c57119. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.197:38987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:00.144032 5458 leader_election.cc:302] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 163649b5fbb140f18dccdf75f608b0f9, 9715934145414f2496a4105e95aa0987; no voters: 269c9061e1a1435e92f79f0b9486dfea, 4d99e1fdda704211b3d101f892c57119, 7ef43ea75ecb46ee905b781c91aed95a
I20240502 01:54:00.144433 6602 raft_consensus.cc:2726] T 74753a745d874fcea0fa9dfc4649aa14 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:00.148514 2691 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:00.223263 6602 raft_consensus.cc:482] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 58dae5dc94444d5e92e6907c5b637879)
I20240502 01:54:00.223471 6602 raft_consensus.cc:504] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 375 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:54:00.224581 6602 leader_election.cc:288] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225)
W20240502 01:54:00.226063 5458 leader_election.cc:334] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353): Network error: Client connection negotiation failed: client connection to 127.31.161.195:39353: connect: Connection refused (error 111)
W20240502 01:54:00.226235 5460 leader_election.cc:334] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): Network error: Client connection negotiation failed: client connection to 127.31.161.193:39307: connect: Connection refused (error 111)
W20240502 01:54:00.226745 5459 leader_election.cc:334] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225): Network error: Client connection negotiation failed: client connection to 127.31.161.196:39225: connect: Connection refused (error 111)
I20240502 01:54:00.226867 5459 leader_election.cc:302] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 9715934145414f2496a4105e95aa0987; no voters: 446c7f9bf31a4acc9dae8daf43330906, 58dae5dc94444d5e92e6907c5b637879, 7ef43ea75ecb46ee905b781c91aed95a
I20240502 01:54:00.227095 6602 raft_consensus.cc:2726] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:00.289572 6593 raft_consensus.cc:482] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 446c7f9bf31a4acc9dae8daf43330906)
I20240502 01:54:00.289820 6593 raft_consensus.cc:504] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:54:00.291157 6593 leader_election.cc:288] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 9715934145414f2496a4105e95aa0987 (127.31.161.200:36271)
I20240502 01:54:00.292042 6602 raft_consensus.cc:482] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 446c7f9bf31a4acc9dae8daf43330906)
I20240502 01:54:00.292066 5523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a07f0fe1964f958ad7ca4a63d054a6" candidate_uuid: "163649b5fbb140f18dccdf75f608b0f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 334 } } ignore_live_leader: false dest_uuid: "9715934145414f2496a4105e95aa0987" is_pre_election: true
I20240502 01:54:00.292295 6602 raft_consensus.cc:504] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
W20240502 01:54:00.292840 3539 leader_election.cc:334] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): Network error: Client connection negotiation failed: client connection to 127.31.161.193:39307: connect: Connection refused (error 111)
W20240502 01:54:00.293326 3537 leader_election.cc:334] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353): Network error: Client connection negotiation failed: client connection to 127.31.161.195:39353: connect: Connection refused (error 111)
I20240502 01:54:00.293424 6602 leader_election.cc:288] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381)
I20240502 01:54:00.293443 3537 leader_election.cc:302] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 163649b5fbb140f18dccdf75f608b0f9; no voters: 446c7f9bf31a4acc9dae8daf43330906, 58dae5dc94444d5e92e6907c5b637879, 9715934145414f2496a4105e95aa0987
I20240502 01:54:00.293761 6593 raft_consensus.cc:2726] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:00.294337 3604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d592edf3298942cd93739aa846f867a4" candidate_uuid: "9715934145414f2496a4105e95aa0987" candidate_term: 2 candidate_status { last_received { term: 1 index: 330 } } ignore_live_leader: false dest_uuid: "163649b5fbb140f18dccdf75f608b0f9" is_pre_election: true
W20240502 01:54:00.294943 5458 leader_election.cc:334] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353): Network error: Client connection negotiation failed: client connection to 127.31.161.195:39353: connect: Connection refused (error 111)
W20240502 01:54:00.295578 5459 leader_election.cc:334] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225): Network error: Client connection negotiation failed: client connection to 127.31.161.196:39225: connect: Connection refused (error 111)
I20240502 01:54:00.295718 5459 leader_election.cc:302] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 9715934145414f2496a4105e95aa0987; no voters: 163649b5fbb140f18dccdf75f608b0f9, 446c7f9bf31a4acc9dae8daf43330906, 7ef43ea75ecb46ee905b781c91aed95a
W20240502 01:54:00.295876 5460 leader_election.cc:334] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): Network error: Client connection negotiation failed: client connection to 127.31.161.193:39307: connect: Connection refused (error 111)
I20240502 01:54:00.295969 6602 raft_consensus.cc:2726] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:54:00.373406 3537 consensus_peers.cc:485] T 0225b684df4f41499f76fe10b72a8861 P 163649b5fbb140f18dccdf75f608b0f9 -> Peer 4d99e1fdda704211b3d101f892c57119 (127.31.161.197:38987): Couldn't send request to peer 4d99e1fdda704211b3d101f892c57119. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.197:38987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:00.417867 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 2568
I20240502 01:54:00.437527 6593 raft_consensus.cc:482] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 446c7f9bf31a4acc9dae8daf43330906)
I20240502 01:54:00.437753 6593 raft_consensus.cc:504] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 330 OBSOLETE_local: false peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:54:00.439162 6593 leader_election.cc:288] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 9715934145414f2496a4105e95aa0987 (127.31.161.200:36271)
I20240502 01:54:00.440062 5523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d592edf3298942cd93739aa846f867a4" candidate_uuid: "163649b5fbb140f18dccdf75f608b0f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 330 } } ignore_live_leader: false dest_uuid: "9715934145414f2496a4105e95aa0987" is_pre_election: true
I20240502 01:54:00.440344 5523 raft_consensus.cc:2454] T d592edf3298942cd93739aa846f867a4 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 163649b5fbb140f18dccdf75f608b0f9 in term 1.
W20240502 01:54:00.440423 3539 leader_election.cc:334] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): Network error: Client connection negotiation failed: client connection to 127.31.161.193:39307: connect: Connection refused (error 111)
W20240502 01:54:00.440773 3537 leader_election.cc:334] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353): Network error: Client connection negotiation failed: client connection to 127.31.161.195:39353: connect: Connection refused (error 111)
W20240502 01:54:00.441452 3538 leader_election.cc:334] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225): Network error: Client connection negotiation failed: client connection to 127.31.161.196:39225: connect: Connection refused (error 111)
I20240502 01:54:00.441555 3538 leader_election.cc:302] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 163649b5fbb140f18dccdf75f608b0f9, 9715934145414f2496a4105e95aa0987; no voters: 446c7f9bf31a4acc9dae8daf43330906, 58dae5dc94444d5e92e6907c5b637879, 7ef43ea75ecb46ee905b781c91aed95a
I20240502 01:54:00.441864 6593 raft_consensus.cc:2726] T d592edf3298942cd93739aa846f867a4 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:00.464984 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 3522
W20240502 01:54:00.465606 3537 leader_election.cc:334] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585): Network error: Client connection negotiation failed: client connection to 127.31.161.198:37585: connect: Connection refused (error 111)
W20240502 01:54:00.466151 5458 leader_election.cc:334] T 4abedcf755774255867d46154f8131c7 P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585): Network error: Client connection negotiation failed: client connection to 127.31.161.198:37585: connect: Connection refused (error 111)
I20240502 01:54:00.470758 3645 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:00.480415 6602 raft_consensus.cc:482] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 446c7f9bf31a4acc9dae8daf43330906)
I20240502 01:54:00.480607 6602 raft_consensus.cc:504] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 334 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "446c7f9bf31a4acc9dae8daf43330906" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 39353 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:54:00.481603 6602 leader_election.cc:288] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353), 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381)
W20240502 01:54:00.483309 5458 leader_election.cc:334] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 446c7f9bf31a4acc9dae8daf43330906 (127.31.161.195:39353): Network error: Client connection negotiation failed: client connection to 127.31.161.195:39353: connect: Connection refused (error 111)
W20240502 01:54:00.483538 5458 leader_election.cc:334] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585): Network error: Client connection negotiation failed: client connection to 127.31.161.198:37585: connect: Connection refused (error 111)
W20240502 01:54:00.483758 5460 leader_election.cc:334] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): Network error: Client connection negotiation failed: client connection to 127.31.161.193:39307: connect: Connection refused (error 111)
I20240502 01:54:00.483858 5460 leader_election.cc:302] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 9715934145414f2496a4105e95aa0987; no voters: 446c7f9bf31a4acc9dae8daf43330906, 58dae5dc94444d5e92e6907c5b637879, 9a3448a425c041258991dc60dd64ae7d
I20240502 01:54:00.484098 6602 raft_consensus.cc:2726] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:00.704926 3604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4a07f0fe1964f958ad7ca4a63d054a6" candidate_uuid: "9715934145414f2496a4105e95aa0987" candidate_term: 2 candidate_status { last_received { term: 1 index: 334 } } ignore_live_leader: false dest_uuid: "163649b5fbb140f18dccdf75f608b0f9" is_pre_election: true
I20240502 01:54:00.704967 6593 raft_consensus.cc:482] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7ef43ea75ecb46ee905b781c91aed95a)
I20240502 01:54:00.705268 6593 raft_consensus.cc:504] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:54:00.705273 3604 raft_consensus.cc:2454] T f4a07f0fe1964f958ad7ca4a63d054a6 P 163649b5fbb140f18dccdf75f608b0f9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9715934145414f2496a4105e95aa0987 in term 1.
I20240502 01:54:00.705570 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 3522
I20240502 01:54:00.715154 6602 raft_consensus.cc:482] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7ef43ea75ecb46ee905b781c91aed95a)
I20240502 01:54:00.715404 6602 raft_consensus.cc:504] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 351 OBSOLETE_local: false peers { permanent_uuid: "58dae5dc94444d5e92e6907c5b637879" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 39307 } } peers { permanent_uuid: "9a3448a425c041258991dc60dd64ae7d" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37585 } } peers { permanent_uuid: "7ef43ea75ecb46ee905b781c91aed95a" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 39225 } } peers { permanent_uuid: "163649b5fbb140f18dccdf75f608b0f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46381 } attrs { promote: false } } peers { permanent_uuid: "9715934145414f2496a4105e95aa0987" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 36271 } attrs { promote: false } }
I20240502 01:54:00.716446 6602 leader_election.cc:288] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307), 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585), 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225), 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381)
W20240502 01:54:00.718094 5458 leader_election.cc:334] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9a3448a425c041258991dc60dd64ae7d (127.31.161.198:37585): Network error: Client connection negotiation failed: client connection to 127.31.161.198:37585: connect: Connection refused (error 111)
W20240502 01:54:00.718326 5459 leader_election.cc:334] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7ef43ea75ecb46ee905b781c91aed95a (127.31.161.196:39225): Network error: Client connection negotiation failed: client connection to 127.31.161.196:39225: connect: Connection refused (error 111)
W20240502 01:54:00.718569 5460 leader_election.cc:334] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 58dae5dc94444d5e92e6907c5b637879 (127.31.161.193:39307): Network error: Client connection negotiation failed: client connection to 127.31.161.193:39307: connect: Connection refused (error 111)
I20240502 01:54:00.718662 5460 leader_election.cc:302] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 9715934145414f2496a4105e95aa0987; no voters: 58dae5dc94444d5e92e6907c5b637879, 7ef43ea75ecb46ee905b781c91aed95a, 9a3448a425c041258991dc60dd64ae7d
I20240502 01:54:00.718930 6602 raft_consensus.cc:2726] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:54:00.746726 5460 connection.cc:535] client connection to 127.31.161.199:46381 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:00.747128 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 5442
W20240502 01:54:00.748610 5460 leader_election.cc:334] T f4a07f0fe1964f958ad7ca4a63d054a6 P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381): Network error: Client connection negotiation failed: client connection to 127.31.161.199:46381: connect: Connection refused (error 111)
W20240502 01:54:00.749017 5460 leader_election.cc:334] T 4220d1a63bd74ad0a5cac5a2f1bb988c P 9715934145414f2496a4105e95aa0987 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 163649b5fbb140f18dccdf75f608b0f9 (127.31.161.199:46381): Network error: Client connection negotiation failed: client connection to 127.31.161.199:46381: connect: Connection refused (error 111)
I20240502 01:54:00.754201 5566 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:00.913161 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 5442
W20240502 01:54:00.936713 1835 connection.cc:535] server connection from 127.31.161.200:50911 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:00.936931 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 1819
I20240502 01:54:00.938004 1880 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:01.119462 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 1819
2024-05-02T01:54:01Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/7 (77398 ms)
[----------] 1 test from RebalanceParamTest (77398 ms total)
[----------] 1 test from TserverAddedDuringRebalancingTest
[ RUN ] TserverAddedDuringRebalancingTest.TserverStarts/1
I20240502 01:54:01.277643 32391 test_util.cc:268] Using random seed: -527795007
I20240502 01:54:01.279140 32391 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:54:01.279191 32391 ts_itest-base.cc:113] --------------
I20240502 01:54:01.279222 32391 ts_itest-base.cc:114] 8 tablet servers
I20240502 01:54:01.279258 32391 ts_itest-base.cc:115] 3 replicas per TS
I20240502 01:54:01.279289 32391 ts_itest-base.cc:116] --------------
2024-05-02T01:54:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:54:01Z Disabled control of system clock
I20240502 01:54:01.327652 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.161.254:36997
--webserver_interface=127.31.161.254
--webserver_port=0
--builtin_ntp_servers=127.31.161.212:43969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.161.254:36997
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 01:54:01.651819 6629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:01.652151 6629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:01.652288 6629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:01.659749 6629 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:01.659906 6629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:54:01.659992 6629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:01.660072 6629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:54:01.660137 6629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:54:01.669188 6629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:43969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.31.161.254:36997
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.161.254:36997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.31.161.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:01.670456 6629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:01.672174 6629 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:01.682492 6634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:01.683113 6637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:01.683789 6635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:01.684051 6629 server_base.cc:1034] running on GCE node
I20240502 01:54:01.684782 6629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:01.686231 6629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:01.687462 6629 hybrid_clock.cc:648] HybridClock initialized: now 1714614841687409 us; error 41 us; skew 500 ppm
I20240502 01:54:01.687801 6629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:01.689985 6629 webserver.cc:460] Webserver started at http://127.31.161.254:33983/ using document root <none> and password file <none>
I20240502 01:54:01.690682 6629 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:01.690800 6629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:01.691134 6629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:01.693768 6629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2a96c230e6a04167a55c5e28a79211e0"
format_stamp: "Formatted at 2024-05-02 01:54:01 on dist-test-slave-06l8"
I20240502 01:54:01.694499 6629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2a96c230e6a04167a55c5e28a79211e0"
format_stamp: "Formatted at 2024-05-02 01:54:01 on dist-test-slave-06l8"
I20240502 01:54:01.699581 6629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 01:54:01.702915 6643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:01.704156 6629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:01.704319 6629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2a96c230e6a04167a55c5e28a79211e0"
format_stamp: "Formatted at 2024-05-02 01:54:01 on dist-test-slave-06l8"
I20240502 01:54:01.704504 6629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:01.720810 6629 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:01.721637 6629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:01.721931 6629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:01.738391 6629 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.254:36997
I20240502 01:54:01.738399 6694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.254:36997 every 8 connection(s)
I20240502 01:54:01.739660 6629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:54:01.742223 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 6629
I20240502 01:54:01.742442 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:54:01.743367 6695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:01.756255 6695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0: Bootstrap starting.
I20240502 01:54:01.759603 6695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:01.760553 6695 log.cc:824] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:01.763100 6695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0: No bootstrap required, opened a new log
I20240502 01:54:01.769286 6695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a96c230e6a04167a55c5e28a79211e0" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 36997 } }
I20240502 01:54:01.769593 6695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:01.769672 6695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a96c230e6a04167a55c5e28a79211e0, State: Initialized, Role: FOLLOWER
I20240502 01:54:01.770215 6695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a96c230e6a04167a55c5e28a79211e0" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 36997 } }
I20240502 01:54:01.770417 6695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:01.770509 6695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:01.770622 6695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:01.772549 6695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a96c230e6a04167a55c5e28a79211e0" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 36997 } }
I20240502 01:54:01.773006 6695 leader_election.cc:302] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a96c230e6a04167a55c5e28a79211e0; no voters:
I20240502 01:54:01.773584 6695 leader_election.cc:288] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:54:01.773708 6700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:01.774725 6700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0 [term 1 LEADER]: Becoming Leader. State: Replica: 2a96c230e6a04167a55c5e28a79211e0, State: Running, Role: LEADER
I20240502 01:54:01.775312 6700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a96c230e6a04167a55c5e28a79211e0" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 36997 } }
I20240502 01:54:01.775506 6695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:54:01.777841 6701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a96c230e6a04167a55c5e28a79211e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a96c230e6a04167a55c5e28a79211e0" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 36997 } } }
I20240502 01:54:01.778195 6701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0 [sys.catalog]: This master's current role is: LEADER
I20240502 01:54:01.778121 6702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a96c230e6a04167a55c5e28a79211e0. Latest consensus state: current_term: 1 leader_uuid: "2a96c230e6a04167a55c5e28a79211e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a96c230e6a04167a55c5e28a79211e0" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 36997 } } }
I20240502 01:54:01.778398 6702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0 [sys.catalog]: This master's current role is: LEADER
I20240502 01:54:01.779970 6709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:54:01.784865 6709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:54:01.792263 6709 catalog_manager.cc:1347] Generated new cluster ID: 514a9534b24942e5ab7b5207fc6672c1
I20240502 01:54:01.792367 6709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:54:01.807454 6709 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:54:01.808620 6709 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:54:01.829823 6709 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2a96c230e6a04167a55c5e28a79211e0: Generated new TSK 0
I20240502 01:54:01.830637 6709 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:54:01.844566 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.193:0
--local_ip_for_outbound_sockets=127.31.161.193
--webserver_interface=127.31.161.193
--webserver_port=0
--tserver_master_addrs=127.31.161.254:36997
--builtin_ntp_servers=127.31.161.212:43969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 01:54:02.176795 6720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:02.177145 6720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:02.177289 6720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:02.184865 6720 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:02.185002 6720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:02.185178 6720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.193
I20240502 01:54:02.194116 6720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:43969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.31.161.193
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.161.254:36997
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.193
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:02.195451 6720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:02.197245 6720 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:02.208734 6725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:02.208766 6729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:02.209398 6727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:02.209627 6720 server_base.cc:1034] running on GCE node
I20240502 01:54:02.210142 6720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:02.210983 6720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:02.212205 6720 hybrid_clock.cc:648] HybridClock initialized: now 1714614842212156 us; error 34 us; skew 500 ppm
I20240502 01:54:02.212499 6720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:02.214718 6720 webserver.cc:460] Webserver started at http://127.31.161.193:45509/ using document root <none> and password file <none>
I20240502 01:54:02.215399 6720 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:02.215493 6720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:02.215771 6720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:02.218349 6720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
format_stamp: "Formatted at 2024-05-02 01:54:02 on dist-test-slave-06l8"
I20240502 01:54:02.219043 6720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
format_stamp: "Formatted at 2024-05-02 01:54:02 on dist-test-slave-06l8"
I20240502 01:54:02.224000 6720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 01:54:02.227211 6735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:02.228466 6720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:02.228596 6720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
format_stamp: "Formatted at 2024-05-02 01:54:02 on dist-test-slave-06l8"
I20240502 01:54:02.228760 6720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:02.257283 6720 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:02.258164 6720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:02.258417 6720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:02.259351 6720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:02.260838 6720 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:02.260926 6720 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:02.261008 6720 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:02.261062 6720 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:02.290818 6720 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.193:45489
I20240502 01:54:02.290879 6849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.193:45489 every 8 connection(s)
I20240502 01:54:02.292163 6720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:54:02.294474 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 6720
I20240502 01:54:02.294680 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:54:02.298079 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.194:0
--local_ip_for_outbound_sockets=127.31.161.194
--webserver_interface=127.31.161.194
--webserver_port=0
--tserver_master_addrs=127.31.161.254:36997
--builtin_ntp_servers=127.31.161.212:43969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:54:02.305110 6850 heartbeater.cc:347] Connected to a master server at 127.31.161.254:36997
I20240502 01:54:02.305471 6850 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:02.306504 6850 heartbeater.cc:511] Master 127.31.161.254:36997 requested a full tablet report, sending...
I20240502 01:54:02.308763 6660 ts_manager.cc:194] Registered new tserver with Master: 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:02.310431 6660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.193:40829
W20240502 01:54:02.628499 6854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:02.628782 6854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:02.628935 6854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:02.636389 6854 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:02.636512 6854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:02.636670 6854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.194
I20240502 01:54:02.646561 6854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:43969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.31.161.194
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.161.254:36997
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.194
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:02.647891 6854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:02.649659 6854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:02.661093 6859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:02.661679 6860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:02.661823 6862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:02.662323 6854 server_base.cc:1034] running on GCE node
I20240502 01:54:02.662858 6854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:02.663754 6854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:02.664983 6854 hybrid_clock.cc:648] HybridClock initialized: now 1714614842664932 us; error 40 us; skew 500 ppm
I20240502 01:54:02.665309 6854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:02.667438 6854 webserver.cc:460] Webserver started at http://127.31.161.194:43799/ using document root <none> and password file <none>
I20240502 01:54:02.668110 6854 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:02.668205 6854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:02.668483 6854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:02.671283 6854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "137e2183ff57411791e9af58dfab0a3c"
format_stamp: "Formatted at 2024-05-02 01:54:02 on dist-test-slave-06l8"
I20240502 01:54:02.671970 6854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "137e2183ff57411791e9af58dfab0a3c"
format_stamp: "Formatted at 2024-05-02 01:54:02 on dist-test-slave-06l8"
I20240502 01:54:02.676985 6854 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20240502 01:54:02.680315 6868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:02.681627 6854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:54:02.681773 6854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "137e2183ff57411791e9af58dfab0a3c"
format_stamp: "Formatted at 2024-05-02 01:54:02 on dist-test-slave-06l8"
I20240502 01:54:02.681954 6854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:02.703344 6854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:02.704218 6854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:02.704535 6854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:02.705543 6854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:02.707077 6854 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:02.707163 6854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:02.707283 6854 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:02.707352 6854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:02.736861 6854 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.194:45315
I20240502 01:54:02.736930 6982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.194:45315 every 8 connection(s)
I20240502 01:54:02.738248 6854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:54:02.740226 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 6854
I20240502 01:54:02.740424 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:54:02.744103 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.195:0
--local_ip_for_outbound_sockets=127.31.161.195
--webserver_interface=127.31.161.195
--webserver_port=0
--tserver_master_addrs=127.31.161.254:36997
--builtin_ntp_servers=127.31.161.212:43969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:54:02.751206 6983 heartbeater.cc:347] Connected to a master server at 127.31.161.254:36997
I20240502 01:54:02.751506 6983 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:02.752301 6983 heartbeater.cc:511] Master 127.31.161.254:36997 requested a full tablet report, sending...
I20240502 01:54:02.753677 6660 ts_manager.cc:194] Registered new tserver with Master: 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:02.754660 6660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.194:44127
W20240502 01:54:03.074792 6986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:03.075057 6986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:03.075232 6986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:03.082798 6986 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:03.082907 6986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:03.083065 6986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.195
I20240502 01:54:03.091877 6986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:43969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.31.161.195
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.161.254:36997
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.195
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:03.093209 6986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:03.094939 6986 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:03.106180 6991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:03.106757 6994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:03.107306 6992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:03.107369 6986 server_base.cc:1034] running on GCE node
I20240502 01:54:03.107926 6986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:03.108870 6986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:03.110057 6986 hybrid_clock.cc:648] HybridClock initialized: now 1714614843109996 us; error 57 us; skew 500 ppm
I20240502 01:54:03.110364 6986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:03.112391 6986 webserver.cc:460] Webserver started at http://127.31.161.195:39619/ using document root <none> and password file <none>
I20240502 01:54:03.113097 6986 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:03.113221 6986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:03.113535 6986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:03.116084 6986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "11c097d7d32a4c22a6385a7465876eb3"
format_stamp: "Formatted at 2024-05-02 01:54:03 on dist-test-slave-06l8"
I20240502 01:54:03.116794 6986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "11c097d7d32a4c22a6385a7465876eb3"
format_stamp: "Formatted at 2024-05-02 01:54:03 on dist-test-slave-06l8"
I20240502 01:54:03.121620 6986 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20240502 01:54:03.124719 7000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:03.125967 6986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 01:54:03.126103 6986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "11c097d7d32a4c22a6385a7465876eb3"
format_stamp: "Formatted at 2024-05-02 01:54:03 on dist-test-slave-06l8"
I20240502 01:54:03.126281 6986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:03.145659 6986 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:03.146557 6986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:03.146835 6986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:03.147770 6986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:03.149258 6986 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:03.149364 6986 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:03.149470 6986 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:03.149540 6986 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:03.179555 6986 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.195:46083
I20240502 01:54:03.179638 7113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.195:46083 every 8 connection(s)
I20240502 01:54:03.180938 6986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:54:03.187206 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 6986
I20240502 01:54:03.187417 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:54:03.191030 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.196:0
--local_ip_for_outbound_sockets=127.31.161.196
--webserver_interface=127.31.161.196
--webserver_port=0
--tserver_master_addrs=127.31.161.254:36997
--builtin_ntp_servers=127.31.161.212:43969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:54:03.194249 7114 heartbeater.cc:347] Connected to a master server at 127.31.161.254:36997
I20240502 01:54:03.194576 7114 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:03.195534 7114 heartbeater.cc:511] Master 127.31.161.254:36997 requested a full tablet report, sending...
I20240502 01:54:03.197033 6660 ts_manager.cc:194] Registered new tserver with Master: 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:03.198045 6660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.195:54021
I20240502 01:54:03.313282 6850 heartbeater.cc:503] Master 127.31.161.254:36997 was elected leader, sending a full tablet report...
W20240502 01:54:03.525672 7117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:03.525946 7117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:03.526083 7117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:03.533376 7117 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:03.533480 7117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:03.533643 7117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.196
I20240502 01:54:03.542830 7117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:43969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.31.161.196
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.161.254:36997
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.196
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:03.544175 7117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:03.545971 7117 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:03.558136 7123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:03.558101 7125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:03.558291 7122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:03.558837 7117 server_base.cc:1034] running on GCE node
I20240502 01:54:03.559470 7117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:03.560417 7117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:03.561609 7117 hybrid_clock.cc:648] HybridClock initialized: now 1714614843561547 us; error 52 us; skew 500 ppm
I20240502 01:54:03.561926 7117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:03.567499 7117 webserver.cc:460] Webserver started at http://127.31.161.196:37299/ using document root <none> and password file <none>
I20240502 01:54:03.568238 7117 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:03.568344 7117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:03.568650 7117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:03.571300 7117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "fc21155d36d5416fa3650f0268a20b17"
format_stamp: "Formatted at 2024-05-02 01:54:03 on dist-test-slave-06l8"
I20240502 01:54:03.572064 7117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "fc21155d36d5416fa3650f0268a20b17"
format_stamp: "Formatted at 2024-05-02 01:54:03 on dist-test-slave-06l8"
I20240502 01:54:03.577102 7117 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20240502 01:54:03.580482 7131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:03.581830 7117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:03.581990 7117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "fc21155d36d5416fa3650f0268a20b17"
format_stamp: "Formatted at 2024-05-02 01:54:03 on dist-test-slave-06l8"
I20240502 01:54:03.582175 7117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:03.603732 7117 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:03.604629 7117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:03.604912 7117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:03.605916 7117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:03.607429 7117 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:03.607525 7117 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:03.607659 7117 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:03.607743 7117 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:03.637172 7117 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.196:38873
I20240502 01:54:03.637231 7243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.196:38873 every 8 connection(s)
I20240502 01:54:03.638468 7117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:54:03.643098 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 7117
I20240502 01:54:03.643321 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:54:03.646642 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.197:0
--local_ip_for_outbound_sockets=127.31.161.197
--webserver_interface=127.31.161.197
--webserver_port=0
--tserver_master_addrs=127.31.161.254:36997
--builtin_ntp_servers=127.31.161.212:43969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:54:03.651571 7244 heartbeater.cc:347] Connected to a master server at 127.31.161.254:36997
I20240502 01:54:03.651851 7244 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:03.652664 7244 heartbeater.cc:511] Master 127.31.161.254:36997 requested a full tablet report, sending...
I20240502 01:54:03.654135 6660 ts_manager.cc:194] Registered new tserver with Master: fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873)
I20240502 01:54:03.655109 6660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.196:41779
I20240502 01:54:03.759395 6983 heartbeater.cc:503] Master 127.31.161.254:36997 was elected leader, sending a full tablet report...
W20240502 01:54:03.978296 7247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:03.978587 7247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:03.978700 7247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:03.986253 7247 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:03.986366 7247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:03.986500 7247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.197
I20240502 01:54:03.995576 7247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:43969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.31.161.197
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.161.254:36997
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.197
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:03.996966 7247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:03.998708 7247 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:04.010260 7252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:04.010807 7253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:04.011268 7255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:04.011487 7247 server_base.cc:1034] running on GCE node
I20240502 01:54:04.012090 7247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:04.012988 7247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:04.014226 7247 hybrid_clock.cc:648] HybridClock initialized: now 1714614844014178 us; error 61 us; skew 500 ppm
I20240502 01:54:04.014537 7247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:04.016635 7247 webserver.cc:460] Webserver started at http://127.31.161.197:41839/ using document root <none> and password file <none>
I20240502 01:54:04.017372 7247 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:04.017491 7247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:04.017799 7247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:04.020267 7247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "3f156b7cfb4f40f481699b257153e5cb"
format_stamp: "Formatted at 2024-05-02 01:54:04 on dist-test-slave-06l8"
I20240502 01:54:04.020979 7247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "3f156b7cfb4f40f481699b257153e5cb"
format_stamp: "Formatted at 2024-05-02 01:54:04 on dist-test-slave-06l8"
I20240502 01:54:04.025969 7247 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20240502 01:54:04.029134 7261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:04.030432 7247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 01:54:04.030577 7247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "3f156b7cfb4f40f481699b257153e5cb"
format_stamp: "Formatted at 2024-05-02 01:54:04 on dist-test-slave-06l8"
I20240502 01:54:04.030757 7247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:04.052999 7247 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:04.053942 7247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:04.054201 7247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:04.055142 7247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:04.056622 7247 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:04.056716 7247 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:04.056821 7247 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:04.056890 7247 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:04.086254 7247 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.197:44451
I20240502 01:54:04.086334 7373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.197:44451 every 8 connection(s)
I20240502 01:54:04.087574 7247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:54:04.097868 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 7247
I20240502 01:54:04.098042 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:54:04.099879 7374 heartbeater.cc:347] Connected to a master server at 127.31.161.254:36997
I20240502 01:54:04.100159 7374 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:04.100937 7374 heartbeater.cc:511] Master 127.31.161.254:36997 requested a full tablet report, sending...
I20240502 01:54:04.101094 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.198:0
--local_ip_for_outbound_sockets=127.31.161.198
--webserver_interface=127.31.161.198
--webserver_port=0
--tserver_master_addrs=127.31.161.254:36997
--builtin_ntp_servers=127.31.161.212:43969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:54:04.102311 6660 ts_manager.cc:194] Registered new tserver with Master: 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197:44451)
I20240502 01:54:04.103058 6660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.197:50543
I20240502 01:54:04.200839 7114 heartbeater.cc:503] Master 127.31.161.254:36997 was elected leader, sending a full tablet report...
W20240502 01:54:04.425127 7377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:04.425420 7377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:04.425556 7377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:04.432843 7377 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:04.432965 7377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:04.433159 7377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.198
I20240502 01:54:04.441990 7377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:43969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.31.161.198
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.161.254:36997
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.198
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:04.443259 7377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:04.444937 7377 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:04.456375 7383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:04.456408 7382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:04.457067 7385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:04.457499 7377 server_base.cc:1034] running on GCE node
I20240502 01:54:04.458102 7377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:04.459043 7377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:04.460239 7377 hybrid_clock.cc:648] HybridClock initialized: now 1714614844460191 us; error 40 us; skew 500 ppm
I20240502 01:54:04.460543 7377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:04.462726 7377 webserver.cc:460] Webserver started at http://127.31.161.198:46173/ using document root <none> and password file <none>
I20240502 01:54:04.463416 7377 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:04.463510 7377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:04.463779 7377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:04.466419 7377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "9e47e5c098b848c9abe1559cbcbb6321"
format_stamp: "Formatted at 2024-05-02 01:54:04 on dist-test-slave-06l8"
I20240502 01:54:04.467129 7377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "9e47e5c098b848c9abe1559cbcbb6321"
format_stamp: "Formatted at 2024-05-02 01:54:04 on dist-test-slave-06l8"
I20240502 01:54:04.472101 7377 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240502 01:54:04.475265 7391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:04.476576 7377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:04.476732 7377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9e47e5c098b848c9abe1559cbcbb6321"
format_stamp: "Formatted at 2024-05-02 01:54:04 on dist-test-slave-06l8"
I20240502 01:54:04.476933 7377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:04.501164 7377 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:04.502040 7377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:04.502303 7377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:04.503269 7377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:04.504827 7377 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:04.504921 7377 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:04.505031 7377 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:04.505122 7377 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:04.534262 7377 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.198:37123
I20240502 01:54:04.534317 7503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.198:37123 every 8 connection(s)
I20240502 01:54:04.535595 7377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:54:04.539223 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 7377
I20240502 01:54:04.539441 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 01:54:04.542948 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.199:0
--local_ip_for_outbound_sockets=127.31.161.199
--webserver_interface=127.31.161.199
--webserver_port=0
--tserver_master_addrs=127.31.161.254:36997
--builtin_ntp_servers=127.31.161.212:43969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:54:04.548596 7504 heartbeater.cc:347] Connected to a master server at 127.31.161.254:36997
I20240502 01:54:04.548880 7504 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:04.549813 7504 heartbeater.cc:511] Master 127.31.161.254:36997 requested a full tablet report, sending...
I20240502 01:54:04.551389 6660 ts_manager.cc:194] Registered new tserver with Master: 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198:37123)
I20240502 01:54:04.552249 6660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.198:59103
I20240502 01:54:04.657449 7244 heartbeater.cc:503] Master 127.31.161.254:36997 was elected leader, sending a full tablet report...
W20240502 01:54:04.880230 7507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:04.880524 7507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:04.880668 7507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:04.888389 7507 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:04.888516 7507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:04.888660 7507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.199
I20240502 01:54:04.897639 7507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:43969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.31.161.199
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.161.254:36997
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.199
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:04.898873 7507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:04.900538 7507 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:04.911422 7512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:04.912055 7513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:04.912679 7515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:04.912919 7507 server_base.cc:1034] running on GCE node
I20240502 01:54:04.913488 7507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:04.914327 7507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:04.915560 7507 hybrid_clock.cc:648] HybridClock initialized: now 1714614844915507 us; error 43 us; skew 500 ppm
I20240502 01:54:04.915863 7507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:04.917927 7507 webserver.cc:460] Webserver started at http://127.31.161.199:38573/ using document root <none> and password file <none>
I20240502 01:54:04.918581 7507 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:04.918689 7507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:04.919006 7507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:04.921526 7507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "c39b0639d5fb4d9da09bf1d67d617fcc"
format_stamp: "Formatted at 2024-05-02 01:54:04 on dist-test-slave-06l8"
I20240502 01:54:04.922217 7507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "c39b0639d5fb4d9da09bf1d67d617fcc"
format_stamp: "Formatted at 2024-05-02 01:54:04 on dist-test-slave-06l8"
I20240502 01:54:04.926957 7507 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240502 01:54:04.930060 7521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:04.931344 7507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 01:54:04.931537 7507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "c39b0639d5fb4d9da09bf1d67d617fcc"
format_stamp: "Formatted at 2024-05-02 01:54:04 on dist-test-slave-06l8"
I20240502 01:54:04.931722 7507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:04.949283 7507 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:04.950089 7507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:04.950347 7507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:04.951253 7507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:04.952702 7507 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:04.952809 7507 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:04.952911 7507 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:04.952977 7507 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:04.984961 7507 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.199:35245
I20240502 01:54:04.985015 7633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.199:35245 every 8 connection(s)
I20240502 01:54:04.986364 7507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:54:04.993566 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 7507
I20240502 01:54:04.993774 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 01:54:04.997205 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.200:0
--local_ip_for_outbound_sockets=127.31.161.200
--webserver_interface=127.31.161.200
--webserver_port=0
--tserver_master_addrs=127.31.161.254:36997
--builtin_ntp_servers=127.31.161.212:43969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:54:04.999230 7634 heartbeater.cc:347] Connected to a master server at 127.31.161.254:36997
I20240502 01:54:04.999526 7634 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:05.000324 7634 heartbeater.cc:511] Master 127.31.161.254:36997 requested a full tablet report, sending...
I20240502 01:54:05.001760 6660 ts_manager.cc:194] Registered new tserver with Master: c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199:35245)
I20240502 01:54:05.002705 6660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.199:38233
I20240502 01:54:05.105796 7374 heartbeater.cc:503] Master 127.31.161.254:36997 was elected leader, sending a full tablet report...
W20240502 01:54:05.333765 7637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:05.334069 7637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:05.334228 7637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:05.341981 7637 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:05.342118 7637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:05.342284 7637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.200
I20240502 01:54:05.351258 7637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:43969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.31.161.200
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.161.254:36997
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.200
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:05.352566 7637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:05.354393 7637 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:05.366796 7642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:05.366843 7643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:05.367420 7645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:05.368216 7637 server_base.cc:1034] running on GCE node
I20240502 01:54:05.368757 7637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:05.369653 7637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:05.370844 7637 hybrid_clock.cc:648] HybridClock initialized: now 1714614845370794 us; error 42 us; skew 500 ppm
I20240502 01:54:05.371147 7637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:05.373245 7637 webserver.cc:460] Webserver started at http://127.31.161.200:38861/ using document root <none> and password file <none>
I20240502 01:54:05.373921 7637 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:05.374029 7637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:05.374332 7637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:05.376838 7637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "48866680637c40368e45ba844efb9298"
format_stamp: "Formatted at 2024-05-02 01:54:05 on dist-test-slave-06l8"
I20240502 01:54:05.377584 7637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "48866680637c40368e45ba844efb9298"
format_stamp: "Formatted at 2024-05-02 01:54:05 on dist-test-slave-06l8"
I20240502 01:54:05.382755 7637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20240502 01:54:05.385866 7651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:05.387290 7637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:54:05.387434 7637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "48866680637c40368e45ba844efb9298"
format_stamp: "Formatted at 2024-05-02 01:54:05 on dist-test-slave-06l8"
I20240502 01:54:05.387609 7637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:05.413805 7637 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:05.414635 7637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:05.414881 7637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:05.415791 7637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:05.417306 7637 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:05.417400 7637 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:05.417505 7637 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:05.417572 7637 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:05.446434 7637 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.200:34863
I20240502 01:54:05.446509 7763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.200:34863 every 8 connection(s)
I20240502 01:54:05.447757 7637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:54:05.451948 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 7637
I20240502 01:54:05.452152 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 01:54:05.460494 7764 heartbeater.cc:347] Connected to a master server at 127.31.161.254:36997
I20240502 01:54:05.460773 7764 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:05.461565 7764 heartbeater.cc:511] Master 127.31.161.254:36997 requested a full tablet report, sending...
I20240502 01:54:05.462836 6660 ts_manager.cc:194] Registered new tserver with Master: 48866680637c40368e45ba844efb9298 (127.31.161.200:34863)
I20240502 01:54:05.463603 6660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.200:37205
I20240502 01:54:05.467813 32391 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 01:54:05.483546 32391 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 01:54:05.483655 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 7247
I20240502 01:54:05.491420 7369 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:05.555204 7504 heartbeater.cc:503] Master 127.31.161.254:36997 was elected leader, sending a full tablet report...
I20240502 01:54:05.573130 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 7247
I20240502 01:54:05.589710 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 7377
I20240502 01:54:05.597419 7499 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:05.678045 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 7377
I20240502 01:54:05.692847 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 7507
I20240502 01:54:05.700479 7629 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:05.780184 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 7507
I20240502 01:54:05.794006 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 7637
I20240502 01:54:05.801997 7759 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:05.880466 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 7637
I20240502 01:54:09.647121 6660 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43140:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:09.701658 7048 tablet_service.cc:1467] Processing CreateTablet for tablet d55da66aad2140d788704b056e476e99 (DEFAULT_TABLE table=rebalance_test_table_0 [id=145808b6c1a445a19f5a84827cd55658]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.701774 7179 tablet_service.cc:1467] Processing CreateTablet for tablet d55da66aad2140d788704b056e476e99 (DEFAULT_TABLE table=rebalance_test_table_0 [id=145808b6c1a445a19f5a84827cd55658]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.703186 7179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55da66aad2140d788704b056e476e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.703375 7048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55da66aad2140d788704b056e476e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.703632 7047 tablet_service.cc:1467] Processing CreateTablet for tablet 98630e8d517a4089b236f32f7616cb00 (DEFAULT_TABLE table=rebalance_test_table_0 [id=145808b6c1a445a19f5a84827cd55658]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.704118 7047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98630e8d517a4089b236f32f7616cb00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.704695 7178 tablet_service.cc:1467] Processing CreateTablet for tablet 39f0a7df4f624eee81c25b41d19d1740 (DEFAULT_TABLE table=rebalance_test_table_0 [id=145808b6c1a445a19f5a84827cd55658]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.705199 7178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f0a7df4f624eee81c25b41d19d1740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.709856 6783 tablet_service.cc:1467] Processing CreateTablet for tablet 98630e8d517a4089b236f32f7616cb00 (DEFAULT_TABLE table=rebalance_test_table_0 [id=145808b6c1a445a19f5a84827cd55658]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.710979 6783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98630e8d517a4089b236f32f7616cb00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.711504 6917 tablet_service.cc:1467] Processing CreateTablet for tablet 98630e8d517a4089b236f32f7616cb00 (DEFAULT_TABLE table=rebalance_test_table_0 [id=145808b6c1a445a19f5a84827cd55658]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.709616 6782 tablet_service.cc:1467] Processing CreateTablet for tablet 39f0a7df4f624eee81c25b41d19d1740 (DEFAULT_TABLE table=rebalance_test_table_0 [id=145808b6c1a445a19f5a84827cd55658]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.712724 6917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98630e8d517a4089b236f32f7616cb00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.713064 6782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f0a7df4f624eee81c25b41d19d1740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.713258 6784 tablet_service.cc:1467] Processing CreateTablet for tablet d55da66aad2140d788704b056e476e99 (DEFAULT_TABLE table=rebalance_test_table_0 [id=145808b6c1a445a19f5a84827cd55658]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.713738 6784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55da66aad2140d788704b056e476e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.713752 6916 tablet_service.cc:1467] Processing CreateTablet for tablet 39f0a7df4f624eee81c25b41d19d1740 (DEFAULT_TABLE table=rebalance_test_table_0 [id=145808b6c1a445a19f5a84827cd55658]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:09.714310 6916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f0a7df4f624eee81c25b41d19d1740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:09.738356 7797 tablet_bootstrap.cc:492] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:09.749315 7795 tablet_bootstrap.cc:492] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c: Bootstrap starting.
I20240502 01:54:09.752552 7795 tablet_bootstrap.cc:654] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.757113 7797 tablet_bootstrap.cc:654] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.758137 7795 log.cc:824] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:09.758627 7797 log.cc:824] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:09.762094 7794 tablet_bootstrap.cc:492] T 98630e8d517a4089b236f32f7616cb00 P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:09.762902 7796 tablet_bootstrap.cc:492] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17: Bootstrap starting.
I20240502 01:54:09.765107 7795 tablet_bootstrap.cc:492] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c: No bootstrap required, opened a new log
I20240502 01:54:09.765466 7794 tablet_bootstrap.cc:654] T 98630e8d517a4089b236f32f7616cb00 P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.765470 7795 ts_tablet_manager.cc:1367] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.002s
I20240502 01:54:09.765738 7797 tablet_bootstrap.cc:492] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:09.766005 7797 ts_tablet_manager.cc:1367] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.004s
I20240502 01:54:09.766475 7796 tablet_bootstrap.cc:654] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.766894 7794 log.cc:824] T 98630e8d517a4089b236f32f7616cb00 P 11c097d7d32a4c22a6385a7465876eb3: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:09.767735 7796 log.cc:824] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:09.774127 7795 raft_consensus.cc:348] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.774678 7795 raft_consensus.cc:374] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.774828 7795 raft_consensus.cc:729] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.775728 7795 consensus_queue.cc:260] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.775772 7797 raft_consensus.cc:348] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.776265 7797 raft_consensus.cc:374] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.776405 7797 raft_consensus.cc:729] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.777318 7797 consensus_queue.cc:260] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.777639 7796 tablet_bootstrap.cc:492] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17: No bootstrap required, opened a new log
I20240502 01:54:09.777963 7796 ts_tablet_manager.cc:1367] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.006s
I20240502 01:54:09.778379 7794 tablet_bootstrap.cc:492] T 98630e8d517a4089b236f32f7616cb00 P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:09.778666 7794 ts_tablet_manager.cc:1367] T 98630e8d517a4089b236f32f7616cb00 P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.011s
I20240502 01:54:09.787111 7794 raft_consensus.cc:348] T 98630e8d517a4089b236f32f7616cb00 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.787045 7796 raft_consensus.cc:348] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.787520 7796 raft_consensus.cc:374] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.787672 7796 raft_consensus.cc:729] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.788211 7794 raft_consensus.cc:374] T 98630e8d517a4089b236f32f7616cb00 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.788372 7794 raft_consensus.cc:729] T 98630e8d517a4089b236f32f7616cb00 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.789017 7795 ts_tablet_manager.cc:1398] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c: Time spent starting tablet: real 0.023s user 0.013s sys 0.006s
I20240502 01:54:09.789703 7795 tablet_bootstrap.cc:492] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c: Bootstrap starting.
I20240502 01:54:09.789848 7796 consensus_queue.cc:260] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.791550 7795 tablet_bootstrap.cc:654] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.789350 7794 consensus_queue.cc:260] T 98630e8d517a4089b236f32f7616cb00 P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.793433 7797 ts_tablet_manager.cc:1398] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.027s user 0.003s sys 0.023s
I20240502 01:54:09.793939 7797 tablet_bootstrap.cc:492] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:09.795729 7797 tablet_bootstrap.cc:654] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
W20240502 01:54:09.799137 6851 tablet.cc:2369] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:54:09.799975 7795 tablet_bootstrap.cc:492] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c: No bootstrap required, opened a new log
I20240502 01:54:09.800231 7795 ts_tablet_manager.cc:1367] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20240502 01:54:09.801573 7795 raft_consensus.cc:348] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.801913 7795 raft_consensus.cc:374] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.802055 7795 raft_consensus.cc:729] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.802459 7796 ts_tablet_manager.cc:1398] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17: Time spent starting tablet: real 0.024s user 0.017s sys 0.000s
I20240502 01:54:09.802436 7795 consensus_queue.cc:260] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.803311 7795 ts_tablet_manager.cc:1398] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:09.804069 7796 tablet_bootstrap.cc:492] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17: Bootstrap starting.
I20240502 01:54:09.805689 7796 tablet_bootstrap.cc:654] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.808045 7794 ts_tablet_manager.cc:1398] T 98630e8d517a4089b236f32f7616cb00 P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.029s user 0.007s sys 0.006s
I20240502 01:54:09.808370 7797 tablet_bootstrap.cc:492] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:09.808475 7794 tablet_bootstrap.cc:492] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:09.808493 7796 tablet_bootstrap.cc:492] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17: No bootstrap required, opened a new log
I20240502 01:54:09.808568 7797 ts_tablet_manager.cc:1367] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20240502 01:54:09.808624 7796 ts_tablet_manager.cc:1367] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:54:09.809728 7796 raft_consensus.cc:348] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.809727 7797 raft_consensus.cc:348] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.809954 7797 raft_consensus.cc:374] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.809995 7796 raft_consensus.cc:374] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.810086 7794 tablet_bootstrap.cc:654] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.810093 7797 raft_consensus.cc:729] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.810122 7796 raft_consensus.cc:729] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.810361 7796 consensus_queue.cc:260] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.810534 7797 consensus_queue.cc:260] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.810952 7796 ts_tablet_manager.cc:1398] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:09.811095 7797 ts_tablet_manager.cc:1398] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:09.811437 7797 tablet_bootstrap.cc:492] T 98630e8d517a4089b236f32f7616cb00 P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:09.812400 7794 tablet_bootstrap.cc:492] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:09.812588 7794 ts_tablet_manager.cc:1367] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:54:09.813099 7797 tablet_bootstrap.cc:654] T 98630e8d517a4089b236f32f7616cb00 P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:09.813649 7794 raft_consensus.cc:348] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.813930 7794 raft_consensus.cc:374] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.814045 7794 raft_consensus.cc:729] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.814282 7794 consensus_queue.cc:260] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.814880 7794 ts_tablet_manager.cc:1398] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:09.816056 7797 tablet_bootstrap.cc:492] T 98630e8d517a4089b236f32f7616cb00 P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:09.816195 7797 ts_tablet_manager.cc:1367] T 98630e8d517a4089b236f32f7616cb00 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 01:54:09.816838 7797 raft_consensus.cc:348] T 98630e8d517a4089b236f32f7616cb00 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.817093 7797 raft_consensus.cc:374] T 98630e8d517a4089b236f32f7616cb00 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:09.817189 7797 raft_consensus.cc:729] T 98630e8d517a4089b236f32f7616cb00 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:09.817416 7797 consensus_queue.cc:260] T 98630e8d517a4089b236f32f7616cb00 P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.817886 7797 ts_tablet_manager.cc:1398] T 98630e8d517a4089b236f32f7616cb00 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:09.831581 7804 raft_consensus.cc:482] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:09.831806 7804 raft_consensus.cc:504] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.833582 7804 leader_election.cc:288] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:09.840137 7068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55da66aad2140d788704b056e476e99" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3" is_pre_election: true
I20240502 01:54:09.840196 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55da66aad2140d788704b056e476e99" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" is_pre_election: true
I20240502 01:54:09.840569 7068 raft_consensus.cc:2454] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 0.
I20240502 01:54:09.840632 6804 raft_consensus.cc:2454] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 0.
I20240502 01:54:09.841207 7133 leader_election.cc:302] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8214b6c4de6d4aa58a22af551eb8dedd, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:09.841554 7804 raft_consensus.cc:2781] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:09.841713 7804 raft_consensus.cc:482] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:09.841812 7804 raft_consensus.cc:3037] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.843741 7804 raft_consensus.cc:504] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.844286 7804 leader_election.cc:288] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 election: Requested vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:09.844664 7068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55da66aad2140d788704b056e476e99" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
I20240502 01:54:09.844743 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d55da66aad2140d788704b056e476e99" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
I20240502 01:54:09.844825 7068 raft_consensus.cc:3037] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.844934 6804 raft_consensus.cc:3037] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.846838 7068 raft_consensus.cc:2454] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 1.
I20240502 01:54:09.846982 6804 raft_consensus.cc:2454] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 1.
I20240502 01:54:09.847208 7134 leader_election.cc:302] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:09.847499 7804 raft_consensus.cc:2781] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:09.847903 7804 raft_consensus.cc:686] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Becoming Leader. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Running, Role: LEADER
I20240502 01:54:09.848325 7804 consensus_queue.cc:237] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.853154 6659 catalog_manager.cc:5484] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc21155d36d5416fa3650f0268a20b17 (127.31.161.196). New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:09.867046 7803 raft_consensus.cc:482] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:09.867210 7803 raft_consensus.cc:504] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.868772 7803 leader_election.cc:288] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:09.876021 6937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f0a7df4f624eee81c25b41d19d1740" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c" is_pre_election: true
I20240502 01:54:09.876021 7199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f0a7df4f624eee81c25b41d19d1740" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17" is_pre_election: true
I20240502 01:54:09.876453 7199 raft_consensus.cc:2454] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 0.
I20240502 01:54:09.876562 6937 raft_consensus.cc:2454] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 0.
I20240502 01:54:09.877005 6737 leader_election.cc:302] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8214b6c4de6d4aa58a22af551eb8dedd, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:09.877398 7803 raft_consensus.cc:2781] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:09.877538 7803 raft_consensus.cc:482] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:09.877631 7803 raft_consensus.cc:3037] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.879626 7803 raft_consensus.cc:504] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.880255 7803 leader_election.cc:288] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 election: Requested vote from peers fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:09.880697 7199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f0a7df4f624eee81c25b41d19d1740" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17"
I20240502 01:54:09.880729 6937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f0a7df4f624eee81c25b41d19d1740" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c"
I20240502 01:54:09.880899 7199 raft_consensus.cc:3037] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.880931 6937 raft_consensus.cc:3037] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.883823 7199 raft_consensus.cc:2454] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 1.
I20240502 01:54:09.883823 6937 raft_consensus.cc:2454] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 1.
I20240502 01:54:09.884227 6738 leader_election.cc:302] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 137e2183ff57411791e9af58dfab0a3c, 8214b6c4de6d4aa58a22af551eb8dedd; no voters:
I20240502 01:54:09.884500 7803 raft_consensus.cc:2781] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:09.884896 7803 raft_consensus.cc:686] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Becoming Leader. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Running, Role: LEADER
I20240502 01:54:09.885366 7803 consensus_queue.cc:237] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.889804 6659 catalog_manager.cc:5484] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193). New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } }
W20240502 01:54:09.893990 7245 tablet.cc:2369] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:54:09.927836 7802 raft_consensus.cc:482] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:09.928035 7802 raft_consensus.cc:504] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.929708 7802 leader_election.cc:288] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:09.936939 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98630e8d517a4089b236f32f7616cb00" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" is_pre_election: true
W20240502 01:54:09.937018 7115 tablet.cc:2369] T 98630e8d517a4089b236f32f7616cb00 P 11c097d7d32a4c22a6385a7465876eb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:54:09.936955 7068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98630e8d517a4089b236f32f7616cb00" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3" is_pre_election: true
I20240502 01:54:09.937211 6804 raft_consensus.cc:2454] T 98630e8d517a4089b236f32f7616cb00 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 0.
I20240502 01:54:09.937253 7068 raft_consensus.cc:2454] T 98630e8d517a4089b236f32f7616cb00 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 0.
I20240502 01:54:09.937707 6870 leader_election.cc:302] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 137e2183ff57411791e9af58dfab0a3c, 8214b6c4de6d4aa58a22af551eb8dedd; no voters:
I20240502 01:54:09.938122 7802 raft_consensus.cc:2781] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:09.938236 7802 raft_consensus.cc:482] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:09.938326 7802 raft_consensus.cc:3037] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.940254 7802 raft_consensus.cc:504] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.940856 7802 leader_election.cc:288] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 election: Requested vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:09.941207 7068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98630e8d517a4089b236f32f7616cb00" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
I20240502 01:54:09.941378 7068 raft_consensus.cc:3037] T 98630e8d517a4089b236f32f7616cb00 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.941352 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98630e8d517a4089b236f32f7616cb00" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
I20240502 01:54:09.941570 6804 raft_consensus.cc:3037] T 98630e8d517a4089b236f32f7616cb00 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:09.943130 7068 raft_consensus.cc:2454] T 98630e8d517a4089b236f32f7616cb00 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 1.
I20240502 01:54:09.943349 6804 raft_consensus.cc:2454] T 98630e8d517a4089b236f32f7616cb00 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 1.
I20240502 01:54:09.943473 6871 leader_election.cc:302] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 137e2183ff57411791e9af58dfab0a3c; no voters:
I20240502 01:54:09.943744 7802 raft_consensus.cc:2781] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:09.944152 7802 raft_consensus.cc:686] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Becoming Leader. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Running, Role: LEADER
I20240502 01:54:09.944591 7802 consensus_queue.cc:237] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:09.948892 6659 catalog_manager.cc:5484] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 137e2183ff57411791e9af58dfab0a3c (127.31.161.194). New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: UNKNOWN } } }
W20240502 01:54:09.994450 6984 tablet.cc:2369] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:54:10.277940 7803 consensus_queue.cc:1035] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:10.285702 7803 consensus_queue.cc:1035] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:10.299427 7823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:10.299728 7823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:10.307099 7823 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 01:54:10.350476 7802 consensus_queue.cc:1035] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:10.357453 7821 consensus_queue.cc:1035] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:10.364341 7804 consensus_queue.cc:1035] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:10.370164 7856 consensus_queue.cc:1035] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
INSERT report
rows total: 2000
time total: 240.084 ms
time per row: 0.120042 ms
I20240502 01:54:10.699548 6653 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43140:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:10.707938 7047 tablet_service.cc:1467] Processing CreateTablet for tablet 70ce4bbee414402aa97c9f3914b9d0c6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9a08c31578ae4224915289c7c289ee7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.708271 6916 tablet_service.cc:1467] Processing CreateTablet for tablet 70ce4bbee414402aa97c9f3914b9d0c6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9a08c31578ae4224915289c7c289ee7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.708451 7047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ce4bbee414402aa97c9f3914b9d0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.708752 6916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ce4bbee414402aa97c9f3914b9d0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.709587 6784 tablet_service.cc:1467] Processing CreateTablet for tablet c303b823d03249f89bb6dfaca3934cc9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9a08c31578ae4224915289c7c289ee7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.709867 7048 tablet_service.cc:1467] Processing CreateTablet for tablet 5acc7c2405464a328b2016bb2293f7e8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9a08c31578ae4224915289c7c289ee7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.710027 6784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c303b823d03249f89bb6dfaca3934cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.710263 7048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5acc7c2405464a328b2016bb2293f7e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.710244 7179 tablet_service.cc:1467] Processing CreateTablet for tablet 70ce4bbee414402aa97c9f3914b9d0c6 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9a08c31578ae4224915289c7c289ee7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.710656 7179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ce4bbee414402aa97c9f3914b9d0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.713620 7046 tablet_service.cc:1467] Processing CreateTablet for tablet c303b823d03249f89bb6dfaca3934cc9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9a08c31578ae4224915289c7c289ee7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.713759 7178 tablet_service.cc:1467] Processing CreateTablet for tablet 5acc7c2405464a328b2016bb2293f7e8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9a08c31578ae4224915289c7c289ee7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.714035 7046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c303b823d03249f89bb6dfaca3934cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.714161 7178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5acc7c2405464a328b2016bb2293f7e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.718147 6917 tablet_service.cc:1467] Processing CreateTablet for tablet 5acc7c2405464a328b2016bb2293f7e8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9a08c31578ae4224915289c7c289ee7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.718312 6915 tablet_service.cc:1467] Processing CreateTablet for tablet c303b823d03249f89bb6dfaca3934cc9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=9a08c31578ae4224915289c7c289ee7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:10.718592 6917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5acc7c2405464a328b2016bb2293f7e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.718904 6915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c303b823d03249f89bb6dfaca3934cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:10.730001 7892 tablet_bootstrap.cc:492] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:10.730594 7894 tablet_bootstrap.cc:492] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17: Bootstrap starting.
I20240502 01:54:10.731418 7893 tablet_bootstrap.cc:492] T c303b823d03249f89bb6dfaca3934cc9 P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:10.731909 7892 tablet_bootstrap.cc:654] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.732390 7894 tablet_bootstrap.cc:654] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.733778 7893 tablet_bootstrap.cc:654] T c303b823d03249f89bb6dfaca3934cc9 P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.740423 7894 tablet_bootstrap.cc:492] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17: No bootstrap required, opened a new log
I20240502 01:54:10.740584 7894 ts_tablet_manager.cc:1367] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20240502 01:54:10.741413 7894 raft_consensus.cc:348] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:10.741621 7894 raft_consensus.cc:374] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.741710 7894 raft_consensus.cc:729] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.741971 7894 consensus_queue.cc:260] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:10.744760 7894 ts_tablet_manager.cc:1398] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:10.747299 7894 tablet_bootstrap.cc:492] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17: Bootstrap starting.
I20240502 01:54:10.748731 7894 tablet_bootstrap.cc:654] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.757846 7892 tablet_bootstrap.cc:492] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:10.758026 7892 ts_tablet_manager.cc:1367] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.009s
I20240502 01:54:10.758973 7892 raft_consensus.cc:348] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:10.759193 7892 raft_consensus.cc:374] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.759301 7892 raft_consensus.cc:729] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.759589 7892 consensus_queue.cc:260] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:10.760309 7892 ts_tablet_manager.cc:1398] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 01:54:10.760758 7893 tablet_bootstrap.cc:492] T c303b823d03249f89bb6dfaca3934cc9 P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:10.760960 7893 ts_tablet_manager.cc:1367] T c303b823d03249f89bb6dfaca3934cc9 P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.000s
I20240502 01:54:10.761121 7898 tablet_bootstrap.cc:492] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c: Bootstrap starting.
I20240502 01:54:10.761838 7893 raft_consensus.cc:348] T c303b823d03249f89bb6dfaca3934cc9 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:10.762096 7893 raft_consensus.cc:374] T c303b823d03249f89bb6dfaca3934cc9 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.762220 7893 raft_consensus.cc:729] T c303b823d03249f89bb6dfaca3934cc9 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.762508 7893 consensus_queue.cc:260] T c303b823d03249f89bb6dfaca3934cc9 P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:10.763236 7893 ts_tablet_manager.cc:1398] T c303b823d03249f89bb6dfaca3934cc9 P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:10.763250 7894 tablet_bootstrap.cc:492] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17: No bootstrap required, opened a new log
I20240502 01:54:10.763429 7894 ts_tablet_manager.cc:1367] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.004s
I20240502 01:54:10.764209 7894 raft_consensus.cc:348] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:10.764434 7894 raft_consensus.cc:374] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.764495 7894 raft_consensus.cc:729] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.764752 7894 consensus_queue.cc:260] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:10.765357 7894 ts_tablet_manager.cc:1398] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:10.765563 7893 tablet_bootstrap.cc:492] T 70ce4bbee414402aa97c9f3914b9d0c6 P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:10.766573 7898 tablet_bootstrap.cc:654] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.767957 7893 tablet_bootstrap.cc:654] T 70ce4bbee414402aa97c9f3914b9d0c6 P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.771821 7893 tablet_bootstrap.cc:492] T 70ce4bbee414402aa97c9f3914b9d0c6 P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:10.771973 7893 ts_tablet_manager.cc:1367] T 70ce4bbee414402aa97c9f3914b9d0c6 P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 01:54:10.772073 7898 tablet_bootstrap.cc:492] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c: No bootstrap required, opened a new log
I20240502 01:54:10.772255 7898 ts_tablet_manager.cc:1367] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.007s
I20240502 01:54:10.772626 7893 raft_consensus.cc:348] T 70ce4bbee414402aa97c9f3914b9d0c6 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:10.772837 7893 raft_consensus.cc:374] T 70ce4bbee414402aa97c9f3914b9d0c6 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.772914 7893 raft_consensus.cc:729] T 70ce4bbee414402aa97c9f3914b9d0c6 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.773058 7898 raft_consensus.cc:348] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:10.773170 7893 consensus_queue.cc:260] T 70ce4bbee414402aa97c9f3914b9d0c6 P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:10.773294 7898 raft_consensus.cc:374] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.773515 7898 raft_consensus.cc:729] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.773625 7893 ts_tablet_manager.cc:1398] T 70ce4bbee414402aa97c9f3914b9d0c6 P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:10.773766 7898 consensus_queue.cc:260] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:10.773928 7893 tablet_bootstrap.cc:492] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:10.775334 7893 tablet_bootstrap.cc:654] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.776698 7898 ts_tablet_manager.cc:1398] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:54:10.779330 7898 tablet_bootstrap.cc:492] T c303b823d03249f89bb6dfaca3934cc9 P 137e2183ff57411791e9af58dfab0a3c: Bootstrap starting.
I20240502 01:54:10.780483 7893 tablet_bootstrap.cc:492] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:10.780663 7898 tablet_bootstrap.cc:654] T c303b823d03249f89bb6dfaca3934cc9 P 137e2183ff57411791e9af58dfab0a3c: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.780840 7893 ts_tablet_manager.cc:1367] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20240502 01:54:10.781558 7893 raft_consensus.cc:348] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:10.781761 7893 raft_consensus.cc:374] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.781852 7893 raft_consensus.cc:729] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.782086 7893 consensus_queue.cc:260] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:10.783044 7893 ts_tablet_manager.cc:1398] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:10.783982 7898 tablet_bootstrap.cc:492] T c303b823d03249f89bb6dfaca3934cc9 P 137e2183ff57411791e9af58dfab0a3c: No bootstrap required, opened a new log
I20240502 01:54:10.784142 7898 ts_tablet_manager.cc:1367] T c303b823d03249f89bb6dfaca3934cc9 P 137e2183ff57411791e9af58dfab0a3c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 01:54:10.784746 7898 raft_consensus.cc:348] T c303b823d03249f89bb6dfaca3934cc9 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:10.784934 7898 raft_consensus.cc:374] T c303b823d03249f89bb6dfaca3934cc9 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.785018 7898 raft_consensus.cc:729] T c303b823d03249f89bb6dfaca3934cc9 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.785315 7898 consensus_queue.cc:260] T c303b823d03249f89bb6dfaca3934cc9 P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:10.786208 7898 ts_tablet_manager.cc:1398] T c303b823d03249f89bb6dfaca3934cc9 P 137e2183ff57411791e9af58dfab0a3c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:10.786448 7898 tablet_bootstrap.cc:492] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c: Bootstrap starting.
I20240502 01:54:10.787912 7898 tablet_bootstrap.cc:654] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:10.789588 7898 tablet_bootstrap.cc:492] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c: No bootstrap required, opened a new log
I20240502 01:54:10.789708 7898 ts_tablet_manager.cc:1367] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:10.790222 7898 raft_consensus.cc:348] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:10.790361 7898 raft_consensus.cc:374] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:10.790445 7898 raft_consensus.cc:729] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Initialized, Role: FOLLOWER
I20240502 01:54:10.790618 7898 consensus_queue.cc:260] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:10.791038 7898 ts_tablet_manager.cc:1398] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:54:10.876310 7817 raft_consensus.cc:482] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:10.876477 7817 raft_consensus.cc:504] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:10.877334 7817 leader_election.cc:288] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:10.878532 6937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c303b823d03249f89bb6dfaca3934cc9" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c" is_pre_election: true
I20240502 01:54:10.878798 6937 raft_consensus.cc:2454] T c303b823d03249f89bb6dfaca3934cc9 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 0.
I20240502 01:54:10.879305 6738 leader_election.cc:302] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 137e2183ff57411791e9af58dfab0a3c, 8214b6c4de6d4aa58a22af551eb8dedd; no voters:
I20240502 01:54:10.879602 7817 raft_consensus.cc:2781] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:10.879725 7817 raft_consensus.cc:482] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:10.879817 7817 raft_consensus.cc:3037] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.882542 7817 raft_consensus.cc:504] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:10.883278 7817 leader_election.cc:288] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 election: Requested vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:10.883824 6937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c303b823d03249f89bb6dfaca3934cc9" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c"
I20240502 01:54:10.884032 6937 raft_consensus.cc:3037] T c303b823d03249f89bb6dfaca3934cc9 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.884927 7068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c303b823d03249f89bb6dfaca3934cc9" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
I20240502 01:54:10.884909 7067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c303b823d03249f89bb6dfaca3934cc9" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3" is_pre_election: true
I20240502 01:54:10.885195 7068 raft_consensus.cc:3037] T c303b823d03249f89bb6dfaca3934cc9 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.886567 6937 raft_consensus.cc:2454] T c303b823d03249f89bb6dfaca3934cc9 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 1.
I20240502 01:54:10.886969 6738 leader_election.cc:302] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 137e2183ff57411791e9af58dfab0a3c, 8214b6c4de6d4aa58a22af551eb8dedd; no voters:
I20240502 01:54:10.887287 7817 raft_consensus.cc:2781] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:10.887300 7068 raft_consensus.cc:2454] T c303b823d03249f89bb6dfaca3934cc9 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 1.
I20240502 01:54:10.887446 7817 raft_consensus.cc:686] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Becoming Leader. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Running, Role: LEADER
I20240502 01:54:10.887820 7817 consensus_queue.cc:237] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:10.890424 6654 catalog_manager.cc:5484] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193). New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:10.954082 7867 raft_consensus.cc:482] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:10.954282 7867 raft_consensus.cc:504] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:10.955139 7867 leader_election.cc:288] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:10.955693 7068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ce4bbee414402aa97c9f3914b9d0c6" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3" is_pre_election: true
I20240502 01:54:10.955983 7068 raft_consensus.cc:2454] T 70ce4bbee414402aa97c9f3914b9d0c6 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 0.
I20240502 01:54:10.956884 7134 leader_election.cc:302] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:10.957248 7867 raft_consensus.cc:2781] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:10.957387 7867 raft_consensus.cc:482] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:10.957499 7867 raft_consensus.cc:3037] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.960296 7867 raft_consensus.cc:504] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:10.960989 7867 leader_election.cc:288] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 election: Requested vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:10.961593 7068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ce4bbee414402aa97c9f3914b9d0c6" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
I20240502 01:54:10.962209 7068 raft_consensus.cc:3037] T 70ce4bbee414402aa97c9f3914b9d0c6 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.963685 6937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ce4bbee414402aa97c9f3914b9d0c6" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c" is_pre_election: true
I20240502 01:54:10.963932 6937 raft_consensus.cc:2454] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 0.
I20240502 01:54:10.963858 6936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ce4bbee414402aa97c9f3914b9d0c6" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c"
I20240502 01:54:10.964335 6936 raft_consensus.cc:3037] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:10.966483 7068 raft_consensus.cc:2454] T 70ce4bbee414402aa97c9f3914b9d0c6 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 1.
I20240502 01:54:10.966805 6936 raft_consensus.cc:2454] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 1.
I20240502 01:54:10.966838 7134 leader_election.cc:302] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:10.967151 7867 raft_consensus.cc:2781] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:10.967283 7867 raft_consensus.cc:686] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Becoming Leader. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Running, Role: LEADER
I20240502 01:54:10.967612 7867 consensus_queue.cc:237] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:10.970233 6654 catalog_manager.cc:5484] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc21155d36d5416fa3650f0268a20b17 (127.31.161.196). New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:11.027727 7867 raft_consensus.cc:482] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:11.027927 7867 raft_consensus.cc:504] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:11.028589 7867 leader_election.cc:288] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:11.029234 7068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5acc7c2405464a328b2016bb2293f7e8" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3" is_pre_election: true
I20240502 01:54:11.029311 6936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5acc7c2405464a328b2016bb2293f7e8" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c" is_pre_election: true
I20240502 01:54:11.029448 7068 raft_consensus.cc:2454] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 0.
I20240502 01:54:11.029592 6936 raft_consensus.cc:2454] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 0.
I20240502 01:54:11.029929 7134 leader_election.cc:302] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:11.030256 7867 raft_consensus.cc:2781] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:11.030373 7867 raft_consensus.cc:482] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:11.030488 7867 raft_consensus.cc:3037] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.032325 7867 raft_consensus.cc:504] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:11.032923 7867 leader_election.cc:288] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 election: Requested vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:11.033430 7068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5acc7c2405464a328b2016bb2293f7e8" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
I20240502 01:54:11.033457 6936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5acc7c2405464a328b2016bb2293f7e8" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c"
I20240502 01:54:11.033597 7068 raft_consensus.cc:3037] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.033761 6936 raft_consensus.cc:3037] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.035476 7068 raft_consensus.cc:2454] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 1.
I20240502 01:54:11.035822 7134 leader_election.cc:302] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:11.036096 6936 raft_consensus.cc:2454] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 1.
I20240502 01:54:11.036101 7867 raft_consensus.cc:2781] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:11.036303 7867 raft_consensus.cc:686] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Becoming Leader. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Running, Role: LEADER
I20240502 01:54:11.036582 7867 consensus_queue.cc:237] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:11.039172 6654 catalog_manager.cc:5484] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc21155d36d5416fa3650f0268a20b17 (127.31.161.196). New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:11.294966 7817 consensus_queue.cc:1035] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:11.313194 7817 consensus_queue.cc:1035] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:54:11.359649 7867 consensus_queue.cc:1035] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:11.365550 7867 consensus_queue.cc:1035] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:11.424265 7911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:11.424552 7911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:11.431902 7911 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 01:54:11.544906 7867 consensus_queue.cc:1035] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:11.551239 7868 consensus_queue.cc:1035] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 207.695 ms
time per row: 0.103847 ms
I20240502 01:54:11.781536 6659 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43140:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:11.791694 7179 tablet_service.cc:1467] Processing CreateTablet for tablet dea582c2912241609405695eed436abe (DEFAULT_TABLE table=rebalance_test_table_2 [id=3316a56a8d6043c6b05160d6e55a33e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.791864 6784 tablet_service.cc:1467] Processing CreateTablet for tablet dea582c2912241609405695eed436abe (DEFAULT_TABLE table=rebalance_test_table_2 [id=3316a56a8d6043c6b05160d6e55a33e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.792192 7179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea582c2912241609405695eed436abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.792359 6784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea582c2912241609405695eed436abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.793558 6916 tablet_service.cc:1467] Processing CreateTablet for tablet dea582c2912241609405695eed436abe (DEFAULT_TABLE table=rebalance_test_table_2 [id=3316a56a8d6043c6b05160d6e55a33e0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.793799 7046 tablet_service.cc:1467] Processing CreateTablet for tablet b984c722354c462882428317c411fd78 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3316a56a8d6043c6b05160d6e55a33e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.793982 6916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea582c2912241609405695eed436abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.794199 7046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b984c722354c462882428317c411fd78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.797518 6783 tablet_service.cc:1467] Processing CreateTablet for tablet b984c722354c462882428317c411fd78 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3316a56a8d6043c6b05160d6e55a33e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.797899 6783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b984c722354c462882428317c411fd78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.798770 7178 tablet_service.cc:1467] Processing CreateTablet for tablet b984c722354c462882428317c411fd78 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3316a56a8d6043c6b05160d6e55a33e0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.799137 7178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b984c722354c462882428317c411fd78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.801470 6782 tablet_service.cc:1467] Processing CreateTablet for tablet f03b37fd2f844f61a39daadc698c9b29 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3316a56a8d6043c6b05160d6e55a33e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.801859 6782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f03b37fd2f844f61a39daadc698c9b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.803645 7979 tablet_bootstrap.cc:492] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:11.805449 7979 tablet_bootstrap.cc:654] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.805519 6917 tablet_service.cc:1467] Processing CreateTablet for tablet f03b37fd2f844f61a39daadc698c9b29 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3316a56a8d6043c6b05160d6e55a33e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.805984 6917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f03b37fd2f844f61a39daadc698c9b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.808403 7983 tablet_bootstrap.cc:492] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17: Bootstrap starting.
I20240502 01:54:11.809684 7177 tablet_service.cc:1467] Processing CreateTablet for tablet f03b37fd2f844f61a39daadc698c9b29 (DEFAULT_TABLE table=rebalance_test_table_2 [id=3316a56a8d6043c6b05160d6e55a33e0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:11.810160 7177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f03b37fd2f844f61a39daadc698c9b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:11.811995 7983 tablet_bootstrap.cc:654] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.812754 7982 tablet_bootstrap.cc:492] T b984c722354c462882428317c411fd78 P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:11.814601 7982 tablet_bootstrap.cc:654] T b984c722354c462882428317c411fd78 P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.817335 7979 tablet_bootstrap.cc:492] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:11.817520 7979 ts_tablet_manager.cc:1367] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.005s
I20240502 01:54:11.818395 7979 raft_consensus.cc:348] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:11.818626 7979 raft_consensus.cc:374] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.818713 7979 raft_consensus.cc:729] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.819010 7979 consensus_queue.cc:260] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:11.819659 7979 ts_tablet_manager.cc:1398] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:11.820883 7983 tablet_bootstrap.cc:492] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17: No bootstrap required, opened a new log
I20240502 01:54:11.821051 7983 ts_tablet_manager.cc:1367] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240502 01:54:11.821906 7983 raft_consensus.cc:348] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:11.822101 7983 raft_consensus.cc:374] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.822190 7983 raft_consensus.cc:729] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.822463 7983 consensus_queue.cc:260] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:11.823072 7983 ts_tablet_manager.cc:1398] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:11.823534 7982 tablet_bootstrap.cc:492] T b984c722354c462882428317c411fd78 P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:11.823688 7982 ts_tablet_manager.cc:1367] T b984c722354c462882428317c411fd78 P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20240502 01:54:11.824738 7982 raft_consensus.cc:348] T b984c722354c462882428317c411fd78 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:11.824967 7982 raft_consensus.cc:374] T b984c722354c462882428317c411fd78 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.825067 7982 raft_consensus.cc:729] T b984c722354c462882428317c411fd78 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.825423 7982 consensus_queue.cc:260] T b984c722354c462882428317c411fd78 P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:11.826124 7982 ts_tablet_manager.cc:1398] T b984c722354c462882428317c411fd78 P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:54:11.829289 7983 tablet_bootstrap.cc:492] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17: Bootstrap starting.
I20240502 01:54:11.831259 7983 tablet_bootstrap.cc:654] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.831957 7979 tablet_bootstrap.cc:492] T b984c722354c462882428317c411fd78 P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:11.833691 7979 tablet_bootstrap.cc:654] T b984c722354c462882428317c411fd78 P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.834652 7980 tablet_bootstrap.cc:492] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c: Bootstrap starting.
I20240502 01:54:11.836022 7980 tablet_bootstrap.cc:654] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.837909 7980 tablet_bootstrap.cc:492] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c: No bootstrap required, opened a new log
I20240502 01:54:11.838064 7980 ts_tablet_manager.cc:1367] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:11.838115 7983 tablet_bootstrap.cc:492] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17: No bootstrap required, opened a new log
I20240502 01:54:11.838268 7983 ts_tablet_manager.cc:1367] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240502 01:54:11.838603 7979 tablet_bootstrap.cc:492] T b984c722354c462882428317c411fd78 P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:11.838770 7979 ts_tablet_manager.cc:1367] T b984c722354c462882428317c411fd78 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 01:54:11.838759 7980 raft_consensus.cc:348] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:11.838949 7980 raft_consensus.cc:374] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.839044 7980 raft_consensus.cc:729] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.839020 7983 raft_consensus.cc:348] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:11.839231 7983 raft_consensus.cc:374] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.839300 7983 raft_consensus.cc:729] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.839326 7980 consensus_queue.cc:260] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:11.839521 7979 raft_consensus.cc:348] T b984c722354c462882428317c411fd78 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:11.839730 7979 raft_consensus.cc:374] T b984c722354c462882428317c411fd78 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.839792 7979 raft_consensus.cc:729] T b984c722354c462882428317c411fd78 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.839994 7980 ts_tablet_manager.cc:1398] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:11.840039 7979 consensus_queue.cc:260] T b984c722354c462882428317c411fd78 P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:11.840631 7980 tablet_bootstrap.cc:492] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c: Bootstrap starting.
I20240502 01:54:11.840669 7979 ts_tablet_manager.cc:1398] T b984c722354c462882428317c411fd78 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:11.839583 7983 consensus_queue.cc:260] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:11.841385 7983 ts_tablet_manager.cc:1398] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:11.842096 7980 tablet_bootstrap.cc:654] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.842736 7979 tablet_bootstrap.cc:492] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:11.843611 7867 raft_consensus.cc:482] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:11.843780 7867 raft_consensus.cc:504] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:11.844357 7979 tablet_bootstrap.cc:654] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.844463 7867 leader_election.cc:288] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:11.844971 7067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b984c722354c462882428317c411fd78" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3" is_pre_election: true
I20240502 01:54:11.845245 7067 raft_consensus.cc:2454] T b984c722354c462882428317c411fd78 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 0.
I20240502 01:54:11.845664 7134 leader_election.cc:302] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:11.845775 6803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b984c722354c462882428317c411fd78" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" is_pre_election: true
I20240502 01:54:11.845966 6803 raft_consensus.cc:2454] T b984c722354c462882428317c411fd78 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 0.
I20240502 01:54:11.846000 7867 raft_consensus.cc:2781] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:11.846087 7867 raft_consensus.cc:482] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:11.846200 7867 raft_consensus.cc:3037] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.848244 7983 tablet_bootstrap.cc:492] T f03b37fd2f844f61a39daadc698c9b29 P fc21155d36d5416fa3650f0268a20b17: Bootstrap starting.
I20240502 01:54:11.848836 7867 raft_consensus.cc:504] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:11.849777 7867 leader_election.cc:288] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 election: Requested vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:11.849824 7983 tablet_bootstrap.cc:654] T f03b37fd2f844f61a39daadc698c9b29 P fc21155d36d5416fa3650f0268a20b17: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:11.850186 6803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b984c722354c462882428317c411fd78" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
I20240502 01:54:11.850396 6803 raft_consensus.cc:3037] T b984c722354c462882428317c411fd78 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.850582 7979 tablet_bootstrap.cc:492] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:11.850746 7979 ts_tablet_manager.cc:1367] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:54:11.851634 7979 raft_consensus.cc:348] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:11.851881 7979 raft_consensus.cc:374] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.851985 7979 raft_consensus.cc:729] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.852227 7979 consensus_queue.cc:260] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:11.852701 6803 raft_consensus.cc:2454] T b984c722354c462882428317c411fd78 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 1.
I20240502 01:54:11.852831 7979 ts_tablet_manager.cc:1398] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:11.853061 7067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b984c722354c462882428317c411fd78" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
I20240502 01:54:11.853310 7067 raft_consensus.cc:3037] T b984c722354c462882428317c411fd78 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.855880 7067 raft_consensus.cc:2454] T b984c722354c462882428317c411fd78 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 1.
I20240502 01:54:11.856432 7133 leader_election.cc:302] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 8214b6c4de6d4aa58a22af551eb8dedd, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:11.856752 7867 raft_consensus.cc:2781] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:11.856948 7867 raft_consensus.cc:686] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Becoming Leader. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Running, Role: LEADER
I20240502 01:54:11.857281 7867 consensus_queue.cc:237] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:11.857800 7983 tablet_bootstrap.cc:492] T f03b37fd2f844f61a39daadc698c9b29 P fc21155d36d5416fa3650f0268a20b17: No bootstrap required, opened a new log
I20240502 01:54:11.857964 7983 ts_tablet_manager.cc:1367] T f03b37fd2f844f61a39daadc698c9b29 P fc21155d36d5416fa3650f0268a20b17: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.004s
I20240502 01:54:11.858734 7983 raft_consensus.cc:348] T f03b37fd2f844f61a39daadc698c9b29 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:11.858968 7983 raft_consensus.cc:374] T f03b37fd2f844f61a39daadc698c9b29 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.859074 7983 raft_consensus.cc:729] T f03b37fd2f844f61a39daadc698c9b29 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.859182 7980 tablet_bootstrap.cc:492] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c: No bootstrap required, opened a new log
I20240502 01:54:11.859335 7983 consensus_queue.cc:260] T f03b37fd2f844f61a39daadc698c9b29 P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:11.859542 7980 ts_tablet_manager.cc:1367] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.005s
I20240502 01:54:11.859851 7983 ts_tablet_manager.cc:1398] T f03b37fd2f844f61a39daadc698c9b29 P fc21155d36d5416fa3650f0268a20b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 01:54:11.860323 7980 raft_consensus.cc:348] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:11.860535 7980 raft_consensus.cc:374] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:11.860261 6659 catalog_manager.cc:5484] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc21155d36d5416fa3650f0268a20b17 (127.31.161.196). New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:11.860610 7980 raft_consensus.cc:729] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Initialized, Role: FOLLOWER
I20240502 01:54:11.860847 7980 consensus_queue.cc:260] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:11.861395 7980 ts_tablet_manager.cc:1398] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:11.862581 7959 raft_consensus.cc:482] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:11.862713 7959 raft_consensus.cc:504] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:11.863461 7959 leader_election.cc:288] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:11.863987 7199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f03b37fd2f844f61a39daadc698c9b29" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17" is_pre_election: true
I20240502 01:54:11.864048 6937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f03b37fd2f844f61a39daadc698c9b29" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c" is_pre_election: true
I20240502 01:54:11.864214 7199 raft_consensus.cc:2454] T f03b37fd2f844f61a39daadc698c9b29 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 0.
I20240502 01:54:11.864239 6937 raft_consensus.cc:2454] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 0.
I20240502 01:54:11.864645 6738 leader_election.cc:302] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 137e2183ff57411791e9af58dfab0a3c, 8214b6c4de6d4aa58a22af551eb8dedd; no voters:
I20240502 01:54:11.865033 7959 raft_consensus.cc:2781] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:11.865177 7959 raft_consensus.cc:482] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:11.865276 7959 raft_consensus.cc:3037] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.867518 7959 raft_consensus.cc:504] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:11.868002 7959 leader_election.cc:288] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 election: Requested vote from peers fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:11.868366 7199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f03b37fd2f844f61a39daadc698c9b29" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17"
I20240502 01:54:11.868434 6937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f03b37fd2f844f61a39daadc698c9b29" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c"
I20240502 01:54:11.868533 7199 raft_consensus.cc:3037] T f03b37fd2f844f61a39daadc698c9b29 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.868647 6937 raft_consensus.cc:3037] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.870313 7199 raft_consensus.cc:2454] T f03b37fd2f844f61a39daadc698c9b29 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 1.
I20240502 01:54:11.870527 6937 raft_consensus.cc:2454] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 1.
I20240502 01:54:11.870648 6737 leader_election.cc:302] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8214b6c4de6d4aa58a22af551eb8dedd, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:11.870908 7959 raft_consensus.cc:2781] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:11.871062 7959 raft_consensus.cc:686] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Becoming Leader. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Running, Role: LEADER
I20240502 01:54:11.871503 7959 consensus_queue.cc:237] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:11.873888 6659 catalog_manager.cc:5484] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193). New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:11.888715 7821 raft_consensus.cc:482] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:11.888898 7821 raft_consensus.cc:504] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:11.889614 7821 leader_election.cc:288] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873), 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:11.890798 6803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea582c2912241609405695eed436abe" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" is_pre_election: true
I20240502 01:54:11.891067 6803 raft_consensus.cc:2454] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 0.
I20240502 01:54:11.891501 6870 leader_election.cc:302] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 137e2183ff57411791e9af58dfab0a3c, 8214b6c4de6d4aa58a22af551eb8dedd; no voters:
I20240502 01:54:11.891794 7821 raft_consensus.cc:2781] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:11.891917 7821 raft_consensus.cc:482] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:11.892019 7821 raft_consensus.cc:3037] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.894351 7821 raft_consensus.cc:504] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:11.894974 7821 leader_election.cc:288] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 election: Requested vote from peers fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873), 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:11.895366 6803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea582c2912241609405695eed436abe" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
I20240502 01:54:11.895551 6803 raft_consensus.cc:3037] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:11.896193 7199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea582c2912241609405695eed436abe" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17" is_pre_election: true
I20240502 01:54:11.896234 7198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea582c2912241609405695eed436abe" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17"
I20240502 01:54:11.896456 7199 raft_consensus.cc:2454] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 0.
I20240502 01:54:11.897818 6803 raft_consensus.cc:2454] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 1.
I20240502 01:54:11.898228 6870 leader_election.cc:302] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 137e2183ff57411791e9af58dfab0a3c, 8214b6c4de6d4aa58a22af551eb8dedd; no voters: fc21155d36d5416fa3650f0268a20b17
I20240502 01:54:11.898546 7821 raft_consensus.cc:2781] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:11.899011 7821 raft_consensus.cc:686] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Becoming Leader. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Running, Role: LEADER
I20240502 01:54:11.899302 7821 consensus_queue.cc:237] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:11.902122 6654 catalog_manager.cc:5484] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 137e2183ff57411791e9af58dfab0a3c (127.31.161.194). New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } }
W20240502 01:54:12.254145 7997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:12.254473 7997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:54:12.258929 7959 consensus_queue.cc:1035] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:12.264043 7997 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:54:12.266135 7959 consensus_queue.cc:1035] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:12.284786 7867 consensus_queue.cc:1035] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:12.289687 7965 consensus_queue.cc:1035] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240502 01:54:12.429394 7198 raft_consensus.cc:3037] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.431747 7198 raft_consensus.cc:1261] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:12.432430 7821 consensus_queue.cc:1035] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:12.433779 6803 raft_consensus.cc:1261] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:12.441263 7821 consensus_queue.cc:1035] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:12.494403 8042 mvcc.cc:205] Tried to move back new op lower bound from 7023062435543052288 to 7023062433381978112. Current Snapshot: MvccSnapshot[applied={T|T < 7023062435543052288}]
I20240502 01:54:12.518492 8040 mvcc.cc:205] Tried to move back new op lower bound from 7023062435543052288 to 7023062433381978112. Current Snapshot: MvccSnapshot[applied={T|T < 7023062435543052288}]
INSERT report
rows total: 2000
time total: 214.232 ms
time per row: 0.107116 ms
I20240502 01:54:12.628497 6659 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43140:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:12.637981 7046 tablet_service.cc:1467] Processing CreateTablet for tablet f6a165cbc26f4defb70866779f83bfdc (DEFAULT_TABLE table=rebalance_test_table_3 [id=76b4b12b25de439c840462959d467432]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.638403 7046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a165cbc26f4defb70866779f83bfdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.639256 7048 tablet_service.cc:1467] Processing CreateTablet for tablet 46b6cd134f7e4b2fb56f4df44ed00f15 (DEFAULT_TABLE table=rebalance_test_table_3 [id=76b4b12b25de439c840462959d467432]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.639405 7047 tablet_service.cc:1467] Processing CreateTablet for tablet 01e98643312b4a2eb80e3bcb3d95674c (DEFAULT_TABLE table=rebalance_test_table_3 [id=76b4b12b25de439c840462959d467432]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.639662 7048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46b6cd134f7e4b2fb56f4df44ed00f15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.640039 7047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e98643312b4a2eb80e3bcb3d95674c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.640877 7179 tablet_service.cc:1467] Processing CreateTablet for tablet 01e98643312b4a2eb80e3bcb3d95674c (DEFAULT_TABLE table=rebalance_test_table_3 [id=76b4b12b25de439c840462959d467432]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.641301 7179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e98643312b4a2eb80e3bcb3d95674c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.637557 6782 tablet_service.cc:1467] Processing CreateTablet for tablet f6a165cbc26f4defb70866779f83bfdc (DEFAULT_TABLE table=rebalance_test_table_3 [id=76b4b12b25de439c840462959d467432]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.643755 6782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a165cbc26f4defb70866779f83bfdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.643790 8061 tablet_bootstrap.cc:492] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:12.645583 8061 tablet_bootstrap.cc:654] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.645591 6916 tablet_service.cc:1467] Processing CreateTablet for tablet f6a165cbc26f4defb70866779f83bfdc (DEFAULT_TABLE table=rebalance_test_table_3 [id=76b4b12b25de439c840462959d467432]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.646035 6916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a165cbc26f4defb70866779f83bfdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.647692 8061 tablet_bootstrap.cc:492] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:12.647845 8061 ts_tablet_manager.cc:1367] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:54:12.648633 8061 raft_consensus.cc:348] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:12.648828 8061 raft_consensus.cc:374] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.648929 8061 raft_consensus.cc:729] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.649209 8061 consensus_queue.cc:260] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:12.649451 7177 tablet_service.cc:1467] Processing CreateTablet for tablet 46b6cd134f7e4b2fb56f4df44ed00f15 (DEFAULT_TABLE table=rebalance_test_table_3 [id=76b4b12b25de439c840462959d467432]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.649817 7177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46b6cd134f7e4b2fb56f4df44ed00f15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.649895 8061 ts_tablet_manager.cc:1398] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:12.650290 8061 tablet_bootstrap.cc:492] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:12.655515 8061 tablet_bootstrap.cc:654] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.657403 6783 tablet_service.cc:1467] Processing CreateTablet for tablet 46b6cd134f7e4b2fb56f4df44ed00f15 (DEFAULT_TABLE table=rebalance_test_table_3 [id=76b4b12b25de439c840462959d467432]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.657866 6783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46b6cd134f7e4b2fb56f4df44ed00f15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.658409 8063 tablet_bootstrap.cc:492] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:12.660008 8063 tablet_bootstrap.cc:654] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.661670 6917 tablet_service.cc:1467] Processing CreateTablet for tablet 01e98643312b4a2eb80e3bcb3d95674c (DEFAULT_TABLE table=rebalance_test_table_3 [id=76b4b12b25de439c840462959d467432]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:12.662112 6917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e98643312b4a2eb80e3bcb3d95674c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:12.675922 8067 tablet_bootstrap.cc:492] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17: Bootstrap starting.
I20240502 01:54:12.679226 8066 tablet_bootstrap.cc:492] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c: Bootstrap starting.
I20240502 01:54:12.681747 8067 tablet_bootstrap.cc:654] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.683378 8066 tablet_bootstrap.cc:654] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.688516 8063 tablet_bootstrap.cc:492] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:12.688685 8063 ts_tablet_manager.cc:1367] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.023s
I20240502 01:54:12.689553 8063 raft_consensus.cc:348] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:12.689769 8063 raft_consensus.cc:374] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.689858 8063 raft_consensus.cc:729] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.690115 8063 consensus_queue.cc:260] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:12.690824 8063 ts_tablet_manager.cc:1398] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:54:12.691136 8063 tablet_bootstrap.cc:492] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:12.692584 8063 tablet_bootstrap.cc:654] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.693598 8066 tablet_bootstrap.cc:492] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c: No bootstrap required, opened a new log
I20240502 01:54:12.693768 8066 ts_tablet_manager.cc:1367] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20240502 01:54:12.696638 8066 raft_consensus.cc:348] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:12.696878 8066 raft_consensus.cc:374] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.698611 8067 tablet_bootstrap.cc:492] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17: No bootstrap required, opened a new log
I20240502 01:54:12.698776 8067 ts_tablet_manager.cc:1367] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.003s
I20240502 01:54:12.699481 8067 raft_consensus.cc:348] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:12.699724 8067 raft_consensus.cc:374] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.699837 8067 raft_consensus.cc:729] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.699846 8061 tablet_bootstrap.cc:492] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:12.700030 8061 ts_tablet_manager.cc:1367] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.050s user 0.027s sys 0.019s
I20240502 01:54:12.700083 8067 consensus_queue.cc:260] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:12.700762 8067 ts_tablet_manager.cc:1398] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:12.700927 8063 tablet_bootstrap.cc:492] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:12.701048 8067 tablet_bootstrap.cc:492] T 46b6cd134f7e4b2fb56f4df44ed00f15 P fc21155d36d5416fa3650f0268a20b17: Bootstrap starting.
I20240502 01:54:12.701104 8063 ts_tablet_manager.cc:1367] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20240502 01:54:12.701169 8061 raft_consensus.cc:348] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:12.701387 8061 raft_consensus.cc:374] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.701458 8061 raft_consensus.cc:729] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.701743 8061 consensus_queue.cc:260] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:12.701850 8063 raft_consensus.cc:348] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:12.702067 8063 raft_consensus.cc:374] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.702136 8063 raft_consensus.cc:729] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.702406 8063 consensus_queue.cc:260] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:12.702576 8067 tablet_bootstrap.cc:654] T 46b6cd134f7e4b2fb56f4df44ed00f15 P fc21155d36d5416fa3650f0268a20b17: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.703040 8063 ts_tablet_manager.cc:1398] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:54:12.703447 8066 raft_consensus.cc:729] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.703789 8066 consensus_queue.cc:260] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:12.704380 8066 ts_tablet_manager.cc:1398] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240502 01:54:12.704675 8066 tablet_bootstrap.cc:492] T f6a165cbc26f4defb70866779f83bfdc P 137e2183ff57411791e9af58dfab0a3c: Bootstrap starting.
I20240502 01:54:12.704738 8061 ts_tablet_manager.cc:1398] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:54:12.705678 8061 tablet_bootstrap.cc:492] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:12.706431 8066 tablet_bootstrap.cc:654] T f6a165cbc26f4defb70866779f83bfdc P 137e2183ff57411791e9af58dfab0a3c: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.707327 8061 tablet_bootstrap.cc:654] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:12.711889 8066 tablet_bootstrap.cc:492] T f6a165cbc26f4defb70866779f83bfdc P 137e2183ff57411791e9af58dfab0a3c: No bootstrap required, opened a new log
I20240502 01:54:12.712064 8066 ts_tablet_manager.cc:1367] T f6a165cbc26f4defb70866779f83bfdc P 137e2183ff57411791e9af58dfab0a3c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 01:54:12.712638 8061 tablet_bootstrap.cc:492] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:12.712786 8061 ts_tablet_manager.cc:1367] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20240502 01:54:12.712783 8066 raft_consensus.cc:348] T f6a165cbc26f4defb70866779f83bfdc P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:12.713021 8066 raft_consensus.cc:374] T f6a165cbc26f4defb70866779f83bfdc P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.713124 8066 raft_consensus.cc:729] T f6a165cbc26f4defb70866779f83bfdc P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.713356 8066 consensus_queue.cc:260] T f6a165cbc26f4defb70866779f83bfdc P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:12.713570 8061 raft_consensus.cc:348] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:12.713805 8061 raft_consensus.cc:374] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.713912 8061 raft_consensus.cc:729] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.714184 8061 consensus_queue.cc:260] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:12.714521 8066 ts_tablet_manager.cc:1398] T f6a165cbc26f4defb70866779f83bfdc P 137e2183ff57411791e9af58dfab0a3c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:12.715451 8061 ts_tablet_manager.cc:1398] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:12.724856 8067 tablet_bootstrap.cc:492] T 46b6cd134f7e4b2fb56f4df44ed00f15 P fc21155d36d5416fa3650f0268a20b17: No bootstrap required, opened a new log
I20240502 01:54:12.725034 8067 ts_tablet_manager.cc:1367] T 46b6cd134f7e4b2fb56f4df44ed00f15 P fc21155d36d5416fa3650f0268a20b17: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.014s
I20240502 01:54:12.725728 8067 raft_consensus.cc:348] T 46b6cd134f7e4b2fb56f4df44ed00f15 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:12.725925 8067 raft_consensus.cc:374] T 46b6cd134f7e4b2fb56f4df44ed00f15 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:12.726004 8067 raft_consensus.cc:729] T 46b6cd134f7e4b2fb56f4df44ed00f15 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Initialized, Role: FOLLOWER
I20240502 01:54:12.726269 8067 consensus_queue.cc:260] T 46b6cd134f7e4b2fb56f4df44ed00f15 P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:12.726776 8067 ts_tablet_manager.cc:1398] T 46b6cd134f7e4b2fb56f4df44ed00f15 P fc21155d36d5416fa3650f0268a20b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:12.793725 7912 raft_consensus.cc:482] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:12.793912 7912 raft_consensus.cc:504] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:12.795408 7912 leader_election.cc:288] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873), 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:12.802000 7199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46b6cd134f7e4b2fb56f4df44ed00f15" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17" is_pre_election: true
I20240502 01:54:12.802225 7199 raft_consensus.cc:2454] T 46b6cd134f7e4b2fb56f4df44ed00f15 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 0.
I20240502 01:54:12.802246 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46b6cd134f7e4b2fb56f4df44ed00f15" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" is_pre_election: true
I20240502 01:54:12.802471 6804 raft_consensus.cc:2454] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 0.
I20240502 01:54:12.802697 7002 leader_election.cc:302] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:12.803035 7912 raft_consensus.cc:2781] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:12.803150 7912 raft_consensus.cc:482] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:12.803236 7912 raft_consensus.cc:3037] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.805191 7912 raft_consensus.cc:504] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:12.805740 7912 leader_election.cc:288] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 election: Requested vote from peers fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873), 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:12.806206 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46b6cd134f7e4b2fb56f4df44ed00f15" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
I20240502 01:54:12.806149 7199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46b6cd134f7e4b2fb56f4df44ed00f15" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17"
I20240502 01:54:12.806370 6804 raft_consensus.cc:3037] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.806362 7199 raft_consensus.cc:3037] T 46b6cd134f7e4b2fb56f4df44ed00f15 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.808305 6804 raft_consensus.cc:2454] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 1.
I20240502 01:54:12.808559 7199 raft_consensus.cc:2454] T 46b6cd134f7e4b2fb56f4df44ed00f15 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 1.
I20240502 01:54:12.808719 7002 leader_election.cc:302] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 8214b6c4de6d4aa58a22af551eb8dedd; no voters:
I20240502 01:54:12.809008 7912 raft_consensus.cc:2781] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:12.809473 7912 raft_consensus.cc:686] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Becoming Leader. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Running, Role: LEADER
I20240502 01:54:12.809980 7912 consensus_queue.cc:237] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:12.813513 6652 catalog_manager.cc:5484] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195). New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:12.830233 7912 raft_consensus.cc:482] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:12.830413 7912 raft_consensus.cc:504] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:12.831153 7912 leader_election.cc:288] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:12.831712 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a165cbc26f4defb70866779f83bfdc" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" is_pre_election: true
I20240502 01:54:12.831971 6804 raft_consensus.cc:2454] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 0.
I20240502 01:54:12.832399 7002 leader_election.cc:302] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 8214b6c4de6d4aa58a22af551eb8dedd; no voters:
I20240502 01:54:12.832703 7912 raft_consensus.cc:2781] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:12.832808 7912 raft_consensus.cc:482] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:12.832921 7912 raft_consensus.cc:3037] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.835310 7912 raft_consensus.cc:504] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:12.835866 7912 leader_election.cc:288] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 election: Requested vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:12.836218 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a165cbc26f4defb70866779f83bfdc" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
I20240502 01:54:12.836403 6804 raft_consensus.cc:3037] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.837467 6937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a165cbc26f4defb70866779f83bfdc" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c" is_pre_election: true
I20240502 01:54:12.837558 6936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a165cbc26f4defb70866779f83bfdc" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c"
I20240502 01:54:12.837724 6937 raft_consensus.cc:2454] T f6a165cbc26f4defb70866779f83bfdc P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 0.
I20240502 01:54:12.838600 6804 raft_consensus.cc:2454] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 1.
I20240502 01:54:12.838953 7002 leader_election.cc:302] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 8214b6c4de6d4aa58a22af551eb8dedd; no voters: 137e2183ff57411791e9af58dfab0a3c
I20240502 01:54:12.839224 7912 raft_consensus.cc:2781] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:12.839382 7912 raft_consensus.cc:686] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Becoming Leader. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Running, Role: LEADER
I20240502 01:54:12.839622 7912 consensus_queue.cc:237] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:12.842171 6652 catalog_manager.cc:5484] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195). New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:12.953691 8049 raft_consensus.cc:482] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:12.953881 8049 raft_consensus.cc:504] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:12.954501 8049 leader_election.cc:288] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873), 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:12.954984 7199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e98643312b4a2eb80e3bcb3d95674c" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17" is_pre_election: true
I20240502 01:54:12.955027 7067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e98643312b4a2eb80e3bcb3d95674c" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3" is_pre_election: true
I20240502 01:54:12.955211 7199 raft_consensus.cc:2454] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 0.
I20240502 01:54:12.955269 7067 raft_consensus.cc:2454] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 0.
I20240502 01:54:12.955683 6870 leader_election.cc:302] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 137e2183ff57411791e9af58dfab0a3c, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:12.955953 8049 raft_consensus.cc:2781] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:12.956065 8049 raft_consensus.cc:482] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:12.956172 8049 raft_consensus.cc:3037] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.958400 8049 raft_consensus.cc:504] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:12.958925 8049 leader_election.cc:288] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 election: Requested vote from peers fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873), 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:12.959436 7199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e98643312b4a2eb80e3bcb3d95674c" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17"
I20240502 01:54:12.959630 7199 raft_consensus.cc:3037] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.959563 7067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e98643312b4a2eb80e3bcb3d95674c" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
I20240502 01:54:12.959829 7067 raft_consensus.cc:3037] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:12.962322 7199 raft_consensus.cc:2454] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 1.
I20240502 01:54:12.962743 6870 leader_election.cc:302] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 137e2183ff57411791e9af58dfab0a3c, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:12.963012 8049 raft_consensus.cc:2781] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:12.963181 8049 raft_consensus.cc:686] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Becoming Leader. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Running, Role: LEADER
I20240502 01:54:12.963166 7067 raft_consensus.cc:2454] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 1.
I20240502 01:54:12.963498 8049 consensus_queue.cc:237] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:12.967310 6652 catalog_manager.cc:5484] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 137e2183ff57411791e9af58dfab0a3c (127.31.161.194). New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:13.195217 7912 consensus_queue.cc:1035] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:13.208192 8085 consensus_queue.cc:1035] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:13.359272 8080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:13.359575 8080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:13.366878 8080 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 01:54:13.431423 8085 consensus_queue.cc:1035] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:13.436105 6937 raft_consensus.cc:3037] T f6a165cbc26f4defb70866779f83bfdc P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:13.439105 8076 consensus_queue.cc:1035] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:13.501900 8049 consensus_queue.cc:1035] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:13.510800 7067 raft_consensus.cc:1261] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:13.511431 8049 consensus_queue.cc:1035] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:13.602207 8083 mvcc.cc:205] Tried to move back new op lower bound from 7023062439974490112 to 7023062437741088768. Current Snapshot: MvccSnapshot[applied={T|T < 7023062439974490112}]
INSERT report
rows total: 2000
time total: 264.16 ms
time per row: 0.13208 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 01:54:14.629310 6655 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43140:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:14.638705 6783 tablet_service.cc:1467] Processing CreateTablet for tablet c3397a96d989416fa2bccdc71c7ea9ec (DEFAULT_TABLE table=rebalance_test_table_4 [id=9231a7576c804559bc4a4c54496d5223]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.639200 6783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3397a96d989416fa2bccdc71c7ea9ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.639391 6782 tablet_service.cc:1467] Processing CreateTablet for tablet a14c3c945b724df5ac3f917cef3186e0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9231a7576c804559bc4a4c54496d5223]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.639753 6782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a14c3c945b724df5ac3f917cef3186e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.641968 7048 tablet_service.cc:1467] Processing CreateTablet for tablet c3397a96d989416fa2bccdc71c7ea9ec (DEFAULT_TABLE table=rebalance_test_table_4 [id=9231a7576c804559bc4a4c54496d5223]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.642076 7046 tablet_service.cc:1467] Processing CreateTablet for tablet 640b6474d1834321881094ed0b11b857 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9231a7576c804559bc4a4c54496d5223]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.642400 7048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3397a96d989416fa2bccdc71c7ea9ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.642601 7046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 640b6474d1834321881094ed0b11b857. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.645887 6917 tablet_service.cc:1467] Processing CreateTablet for tablet c3397a96d989416fa2bccdc71c7ea9ec (DEFAULT_TABLE table=rebalance_test_table_4 [id=9231a7576c804559bc4a4c54496d5223]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.645896 6916 tablet_service.cc:1467] Processing CreateTablet for tablet a14c3c945b724df5ac3f917cef3186e0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9231a7576c804559bc4a4c54496d5223]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.646385 6917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3397a96d989416fa2bccdc71c7ea9ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.648916 8144 tablet_bootstrap.cc:492] T c3397a96d989416fa2bccdc71c7ea9ec P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:14.649439 6784 tablet_service.cc:1467] Processing CreateTablet for tablet 640b6474d1834321881094ed0b11b857 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9231a7576c804559bc4a4c54496d5223]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.649864 6784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 640b6474d1834321881094ed0b11b857. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.651974 8144 tablet_bootstrap.cc:654] T c3397a96d989416fa2bccdc71c7ea9ec P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.652565 6916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a14c3c945b724df5ac3f917cef3186e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.653508 7179 tablet_service.cc:1467] Processing CreateTablet for tablet a14c3c945b724df5ac3f917cef3186e0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9231a7576c804559bc4a4c54496d5223]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.653963 7179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a14c3c945b724df5ac3f917cef3186e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.657444 7177 tablet_service.cc:1467] Processing CreateTablet for tablet 640b6474d1834321881094ed0b11b857 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9231a7576c804559bc4a4c54496d5223]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:14.657794 7177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 640b6474d1834321881094ed0b11b857. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:14.668638 8144 tablet_bootstrap.cc:492] T c3397a96d989416fa2bccdc71c7ea9ec P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:14.668793 8144 ts_tablet_manager.cc:1367] T c3397a96d989416fa2bccdc71c7ea9ec P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.005s
I20240502 01:54:14.669560 8144 raft_consensus.cc:348] T c3397a96d989416fa2bccdc71c7ea9ec P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:14.669799 8144 raft_consensus.cc:374] T c3397a96d989416fa2bccdc71c7ea9ec P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.669888 8144 raft_consensus.cc:729] T c3397a96d989416fa2bccdc71c7ea9ec P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.670343 8144 consensus_queue.cc:260] T c3397a96d989416fa2bccdc71c7ea9ec P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:14.670866 8144 ts_tablet_manager.cc:1398] T c3397a96d989416fa2bccdc71c7ea9ec P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:14.671161 8144 tablet_bootstrap.cc:492] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:14.672614 8146 tablet_bootstrap.cc:492] T c3397a96d989416fa2bccdc71c7ea9ec P 137e2183ff57411791e9af58dfab0a3c: Bootstrap starting.
I20240502 01:54:14.674881 8144 tablet_bootstrap.cc:654] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.675096 8146 tablet_bootstrap.cc:654] T c3397a96d989416fa2bccdc71c7ea9ec P 137e2183ff57411791e9af58dfab0a3c: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.676154 8148 tablet_bootstrap.cc:492] T a14c3c945b724df5ac3f917cef3186e0 P fc21155d36d5416fa3650f0268a20b17: Bootstrap starting.
I20240502 01:54:14.680172 8148 tablet_bootstrap.cc:654] T a14c3c945b724df5ac3f917cef3186e0 P fc21155d36d5416fa3650f0268a20b17: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.683602 8146 tablet_bootstrap.cc:492] T c3397a96d989416fa2bccdc71c7ea9ec P 137e2183ff57411791e9af58dfab0a3c: No bootstrap required, opened a new log
I20240502 01:54:14.683756 8146 ts_tablet_manager.cc:1367] T c3397a96d989416fa2bccdc71c7ea9ec P 137e2183ff57411791e9af58dfab0a3c: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20240502 01:54:14.684562 8146 raft_consensus.cc:348] T c3397a96d989416fa2bccdc71c7ea9ec P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:14.684792 8146 raft_consensus.cc:374] T c3397a96d989416fa2bccdc71c7ea9ec P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.684878 8146 raft_consensus.cc:729] T c3397a96d989416fa2bccdc71c7ea9ec P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.685192 8146 consensus_queue.cc:260] T c3397a96d989416fa2bccdc71c7ea9ec P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:14.685806 8146 ts_tablet_manager.cc:1398] T c3397a96d989416fa2bccdc71c7ea9ec P 137e2183ff57411791e9af58dfab0a3c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:14.686278 8146 tablet_bootstrap.cc:492] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c: Bootstrap starting.
I20240502 01:54:14.686529 8145 tablet_bootstrap.cc:492] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:14.686740 8144 tablet_bootstrap.cc:492] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:14.686897 8144 ts_tablet_manager.cc:1367] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.001s
I20240502 01:54:14.687894 8146 tablet_bootstrap.cc:654] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.688079 8144 raft_consensus.cc:348] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:14.688309 8144 raft_consensus.cc:374] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.688392 8144 raft_consensus.cc:729] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.688761 8145 tablet_bootstrap.cc:654] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.690748 8144 consensus_queue.cc:260] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:14.691481 8144 ts_tablet_manager.cc:1398] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:54:14.691910 8148 tablet_bootstrap.cc:492] T a14c3c945b724df5ac3f917cef3186e0 P fc21155d36d5416fa3650f0268a20b17: No bootstrap required, opened a new log
I20240502 01:54:14.692117 8148 ts_tablet_manager.cc:1367] T a14c3c945b724df5ac3f917cef3186e0 P fc21155d36d5416fa3650f0268a20b17: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240502 01:54:14.692955 8148 raft_consensus.cc:348] T a14c3c945b724df5ac3f917cef3186e0 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:14.693260 8148 raft_consensus.cc:374] T a14c3c945b724df5ac3f917cef3186e0 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.693313 8144 tablet_bootstrap.cc:492] T 640b6474d1834321881094ed0b11b857 P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:14.693431 8148 raft_consensus.cc:729] T a14c3c945b724df5ac3f917cef3186e0 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.693719 8148 consensus_queue.cc:260] T a14c3c945b724df5ac3f917cef3186e0 P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:14.694325 8148 ts_tablet_manager.cc:1398] T a14c3c945b724df5ac3f917cef3186e0 P fc21155d36d5416fa3650f0268a20b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:14.694672 8148 tablet_bootstrap.cc:492] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17: Bootstrap starting.
I20240502 01:54:14.696321 8148 tablet_bootstrap.cc:654] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.697599 8145 tablet_bootstrap.cc:492] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:14.697758 8145 ts_tablet_manager.cc:1367] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20240502 01:54:14.698520 8145 raft_consensus.cc:348] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:14.698732 8145 raft_consensus.cc:374] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.698838 8145 raft_consensus.cc:729] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.699092 8145 consensus_queue.cc:260] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:14.701208 8145 ts_tablet_manager.cc:1398] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:14.701489 8145 tablet_bootstrap.cc:492] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:14.703281 8144 tablet_bootstrap.cc:654] T 640b6474d1834321881094ed0b11b857 P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.706784 8146 tablet_bootstrap.cc:492] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c: No bootstrap required, opened a new log
I20240502 01:54:14.706900 8145 tablet_bootstrap.cc:654] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:14.706956 8146 ts_tablet_manager.cc:1367] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.010s
I20240502 01:54:14.707176 8148 tablet_bootstrap.cc:492] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17: No bootstrap required, opened a new log
I20240502 01:54:14.707316 8148 ts_tablet_manager.cc:1367] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20240502 01:54:14.707715 8146 raft_consensus.cc:348] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:14.707937 8146 raft_consensus.cc:374] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.708009 8146 raft_consensus.cc:729] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.708009 8148 raft_consensus.cc:348] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:14.708194 8148 raft_consensus.cc:374] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.708261 8148 raft_consensus.cc:729] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.708269 8146 consensus_queue.cc:260] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:14.708498 8148 consensus_queue.cc:260] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:14.708998 8148 ts_tablet_manager.cc:1398] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:14.709008 8146 ts_tablet_manager.cc:1398] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:14.709546 8144 tablet_bootstrap.cc:492] T 640b6474d1834321881094ed0b11b857 P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:14.709686 8144 ts_tablet_manager.cc:1367] T 640b6474d1834321881094ed0b11b857 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.001s
I20240502 01:54:14.710254 8144 raft_consensus.cc:348] T 640b6474d1834321881094ed0b11b857 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:14.710412 8144 raft_consensus.cc:374] T 640b6474d1834321881094ed0b11b857 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.710494 8144 raft_consensus.cc:729] T 640b6474d1834321881094ed0b11b857 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.710706 8144 consensus_queue.cc:260] T 640b6474d1834321881094ed0b11b857 P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:14.711184 8144 ts_tablet_manager.cc:1398] T 640b6474d1834321881094ed0b11b857 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:54:14.712371 8145 tablet_bootstrap.cc:492] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:14.712532 8145 ts_tablet_manager.cc:1367] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20240502 01:54:14.713313 8145 raft_consensus.cc:348] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:14.713526 8145 raft_consensus.cc:374] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:14.713610 8145 raft_consensus.cc:729] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:14.713847 8145 consensus_queue.cc:260] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:14.714324 8145 ts_tablet_manager.cc:1398] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:14.745687 7912 raft_consensus.cc:482] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:14.745891 7912 raft_consensus.cc:504] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:14.747119 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640b6474d1834321881094ed0b11b857" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" is_pre_election: true
I20240502 01:54:14.747223 7912 leader_election.cc:288] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873), 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:14.747339 6804 raft_consensus.cc:2454] T 640b6474d1834321881094ed0b11b857 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 0.
I20240502 01:54:14.747375 7199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640b6474d1834321881094ed0b11b857" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17" is_pre_election: true
I20240502 01:54:14.747586 7199 raft_consensus.cc:2454] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 0.
I20240502 01:54:14.747809 7002 leader_election.cc:302] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 8214b6c4de6d4aa58a22af551eb8dedd; no voters:
I20240502 01:54:14.748098 8157 raft_consensus.cc:2781] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:14.748210 8157 raft_consensus.cc:482] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:14.748301 8157 raft_consensus.cc:3037] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.750871 7912 raft_consensus.cc:482] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:14.750999 7912 raft_consensus.cc:504] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:14.751119 8157 raft_consensus.cc:504] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:14.751632 7912 leader_election.cc:288] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:14.751737 8157 leader_election.cc:288] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 election: Requested vote from peers fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873), 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:14.752224 7199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640b6474d1834321881094ed0b11b857" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17"
I20240502 01:54:14.752216 6937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3397a96d989416fa2bccdc71c7ea9ec" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c" is_pre_election: true
I20240502 01:54:14.752267 6803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640b6474d1834321881094ed0b11b857" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
I20240502 01:54:14.752416 7199 raft_consensus.cc:3037] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.752425 6937 raft_consensus.cc:2454] T c3397a96d989416fa2bccdc71c7ea9ec P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 0.
I20240502 01:54:14.752475 6803 raft_consensus.cc:3037] T 640b6474d1834321881094ed0b11b857 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.752841 7003 leader_election.cc:302] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 137e2183ff57411791e9af58dfab0a3c; no voters:
I20240502 01:54:14.753291 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3397a96d989416fa2bccdc71c7ea9ec" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" is_pre_election: true
I20240502 01:54:14.753479 6804 raft_consensus.cc:2454] T c3397a96d989416fa2bccdc71c7ea9ec P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 0.
I20240502 01:54:14.753713 8157 raft_consensus.cc:2781] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:14.753806 8157 raft_consensus.cc:482] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:14.753885 8157 raft_consensus.cc:3037] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.755087 7199 raft_consensus.cc:2454] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 1.
I20240502 01:54:14.755193 6803 raft_consensus.cc:2454] T 640b6474d1834321881094ed0b11b857 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 1.
I20240502 01:54:14.755810 7002 leader_election.cc:302] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 8214b6c4de6d4aa58a22af551eb8dedd; no voters:
I20240502 01:54:14.756294 7912 raft_consensus.cc:2781] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:14.756295 8157 raft_consensus.cc:504] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:14.756773 7912 raft_consensus.cc:686] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Becoming Leader. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Running, Role: LEADER
I20240502 01:54:14.757063 8157 leader_election.cc:288] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 election: Requested vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:14.757086 7912 consensus_queue.cc:237] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:14.757386 6937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3397a96d989416fa2bccdc71c7ea9ec" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c"
I20240502 01:54:14.757556 6937 raft_consensus.cc:3037] T c3397a96d989416fa2bccdc71c7ea9ec P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.757555 6803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3397a96d989416fa2bccdc71c7ea9ec" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
I20240502 01:54:14.757740 6803 raft_consensus.cc:3037] T c3397a96d989416fa2bccdc71c7ea9ec P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.759953 6937 raft_consensus.cc:2454] T c3397a96d989416fa2bccdc71c7ea9ec P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 1.
I20240502 01:54:14.760151 6803 raft_consensus.cc:2454] T c3397a96d989416fa2bccdc71c7ea9ec P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 1.
I20240502 01:54:14.760529 7002 leader_election.cc:302] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 8214b6c4de6d4aa58a22af551eb8dedd; no voters:
I20240502 01:54:14.760774 7912 raft_consensus.cc:2781] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:14.760915 7912 raft_consensus.cc:686] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Becoming Leader. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Running, Role: LEADER
I20240502 01:54:14.761163 7912 consensus_queue.cc:237] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:14.761448 6659 catalog_manager.cc:5484] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195). New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:14.765759 6659 catalog_manager.cc:5484] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195). New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:14.853178 7959 raft_consensus.cc:482] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:14.853411 7959 raft_consensus.cc:504] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:14.854118 7959 leader_election.cc:288] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:14.854648 7199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14c3c945b724df5ac3f917cef3186e0" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17" is_pre_election: true
I20240502 01:54:14.854910 7199 raft_consensus.cc:2454] T a14c3c945b724df5ac3f917cef3186e0 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 0.
I20240502 01:54:14.854842 6937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14c3c945b724df5ac3f917cef3186e0" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c" is_pre_election: true
I20240502 01:54:14.855111 6937 raft_consensus.cc:2454] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 0.
I20240502 01:54:14.855329 6737 leader_election.cc:302] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8214b6c4de6d4aa58a22af551eb8dedd, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:14.855671 7959 raft_consensus.cc:2781] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:14.855756 7959 raft_consensus.cc:482] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:14.855854 7959 raft_consensus.cc:3037] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.857743 7959 raft_consensus.cc:504] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:14.858304 7959 leader_election.cc:288] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 election: Requested vote from peers fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:14.858682 7199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14c3c945b724df5ac3f917cef3186e0" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17"
I20240502 01:54:14.858717 6937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a14c3c945b724df5ac3f917cef3186e0" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c"
I20240502 01:54:14.858870 7199 raft_consensus.cc:3037] T a14c3c945b724df5ac3f917cef3186e0 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.858888 6937 raft_consensus.cc:3037] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:14.860674 7199 raft_consensus.cc:2454] T a14c3c945b724df5ac3f917cef3186e0 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 1.
I20240502 01:54:14.860708 6937 raft_consensus.cc:2454] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 1.
I20240502 01:54:14.861049 6737 leader_election.cc:302] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8214b6c4de6d4aa58a22af551eb8dedd, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:14.861332 7959 raft_consensus.cc:2781] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:14.861676 7959 raft_consensus.cc:686] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Becoming Leader. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Running, Role: LEADER
I20240502 01:54:14.861944 7959 consensus_queue.cc:237] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:14.864524 6659 catalog_manager.cc:5484] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193). New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:15.214255 7912 consensus_queue.cc:1035] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:15.219792 8157 consensus_queue.cc:1035] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:15.222486 8163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:15.222925 8163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:15.233068 8163 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:54:15.235229 8157 consensus_queue.cc:1035] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:15.241271 8158 consensus_queue.cc:1035] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240502 01:54:15.337718 7959 consensus_queue.cc:1035] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:15.347503 7959 consensus_queue.cc:1035] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:15.431216 6738 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:45315, user_credentials={real_user=slave}} blocked reactor thread for 50933.4us
INSERT report
rows total: 2000
time total: 225.629 ms
time per row: 0.112814 ms
I20240502 01:54:15.606885 6656 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43140:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:15.615712 7046 tablet_service.cc:1467] Processing CreateTablet for tablet 482f77b65a254ed3aec717c06698ffa7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=abae7d7cdd254582beebaba1abe05ac1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.616174 7046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 482f77b65a254ed3aec717c06698ffa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.616976 7177 tablet_service.cc:1467] Processing CreateTablet for tablet 76fb5e1f69284140a6c6da264a124b3e (DEFAULT_TABLE table=rebalance_test_table_5 [id=abae7d7cdd254582beebaba1abe05ac1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.617152 6784 tablet_service.cc:1467] Processing CreateTablet for tablet d1f4c94a105741339ba69e641244ac18 (DEFAULT_TABLE table=rebalance_test_table_5 [id=abae7d7cdd254582beebaba1abe05ac1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.617152 6917 tablet_service.cc:1467] Processing CreateTablet for tablet 482f77b65a254ed3aec717c06698ffa7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=abae7d7cdd254582beebaba1abe05ac1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.617425 7177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76fb5e1f69284140a6c6da264a124b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.617573 6784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f4c94a105741339ba69e641244ac18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.617573 6917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 482f77b65a254ed3aec717c06698ffa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.621474 7179 tablet_service.cc:1467] Processing CreateTablet for tablet 482f77b65a254ed3aec717c06698ffa7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=abae7d7cdd254582beebaba1abe05ac1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.621866 7179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 482f77b65a254ed3aec717c06698ffa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.624084 6916 tablet_service.cc:1467] Processing CreateTablet for tablet 76fb5e1f69284140a6c6da264a124b3e (DEFAULT_TABLE table=rebalance_test_table_5 [id=abae7d7cdd254582beebaba1abe05ac1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.624552 6916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76fb5e1f69284140a6c6da264a124b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.625950 7048 tablet_service.cc:1467] Processing CreateTablet for tablet 76fb5e1f69284140a6c6da264a124b3e (DEFAULT_TABLE table=rebalance_test_table_5 [id=abae7d7cdd254582beebaba1abe05ac1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.626370 7048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76fb5e1f69284140a6c6da264a124b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.628652 7047 tablet_service.cc:1467] Processing CreateTablet for tablet d1f4c94a105741339ba69e641244ac18 (DEFAULT_TABLE table=rebalance_test_table_5 [id=abae7d7cdd254582beebaba1abe05ac1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.629928 6915 tablet_service.cc:1467] Processing CreateTablet for tablet d1f4c94a105741339ba69e641244ac18 (DEFAULT_TABLE table=rebalance_test_table_5 [id=abae7d7cdd254582beebaba1abe05ac1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:15.630334 6915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f4c94a105741339ba69e641244ac18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.629106 7047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f4c94a105741339ba69e641244ac18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:15.633474 8219 tablet_bootstrap.cc:492] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17: Bootstrap starting.
I20240502 01:54:15.635159 8219 tablet_bootstrap.cc:654] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.635423 8221 tablet_bootstrap.cc:492] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:15.637347 8221 tablet_bootstrap.cc:654] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.638818 8220 tablet_bootstrap.cc:492] T 482f77b65a254ed3aec717c06698ffa7 P 137e2183ff57411791e9af58dfab0a3c: Bootstrap starting.
I20240502 01:54:15.640558 8220 tablet_bootstrap.cc:654] T 482f77b65a254ed3aec717c06698ffa7 P 137e2183ff57411791e9af58dfab0a3c: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.642650 8219 tablet_bootstrap.cc:492] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17: No bootstrap required, opened a new log
I20240502 01:54:15.642827 8219 ts_tablet_manager.cc:1367] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240502 01:54:15.643705 8219 raft_consensus.cc:348] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:15.643949 8219 raft_consensus.cc:374] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.644042 8219 raft_consensus.cc:729] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.644318 8219 consensus_queue.cc:260] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:15.644632 8081 raft_consensus.cc:482] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:15.644790 8081 raft_consensus.cc:504] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:15.645653 8081 leader_election.cc:288] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315), 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:15.646687 7067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fb5e1f69284140a6c6da264a124b3e" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3" is_pre_election: true
W20240502 01:54:15.647825 7134 leader_election.cc:341] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:54:15.650916 7134 leader_election.cc:341] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315): Not found: Tablet not found: 76fb5e1f69284140a6c6da264a124b3e
I20240502 01:54:15.651067 7134 leader_election.cc:302] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fc21155d36d5416fa3650f0268a20b17; no voters: 11c097d7d32a4c22a6385a7465876eb3, 137e2183ff57411791e9af58dfab0a3c
I20240502 01:54:15.651613 8081 raft_consensus.cc:2726] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:15.654464 8219 ts_tablet_manager.cc:1398] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20240502 01:54:15.654873 8219 tablet_bootstrap.cc:492] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17: Bootstrap starting.
I20240502 01:54:15.656739 8219 tablet_bootstrap.cc:654] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.659289 8226 tablet_bootstrap.cc:492] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:15.662683 8226 tablet_bootstrap.cc:654] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.666218 8220 tablet_bootstrap.cc:492] T 482f77b65a254ed3aec717c06698ffa7 P 137e2183ff57411791e9af58dfab0a3c: No bootstrap required, opened a new log
I20240502 01:54:15.666407 8220 ts_tablet_manager.cc:1367] T 482f77b65a254ed3aec717c06698ffa7 P 137e2183ff57411791e9af58dfab0a3c: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.003s
I20240502 01:54:15.667304 8220 raft_consensus.cc:348] T 482f77b65a254ed3aec717c06698ffa7 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:15.667526 8220 raft_consensus.cc:374] T 482f77b65a254ed3aec717c06698ffa7 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.667605 8220 raft_consensus.cc:729] T 482f77b65a254ed3aec717c06698ffa7 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.667847 8220 consensus_queue.cc:260] T 482f77b65a254ed3aec717c06698ffa7 P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:15.668392 8220 ts_tablet_manager.cc:1398] T 482f77b65a254ed3aec717c06698ffa7 P 137e2183ff57411791e9af58dfab0a3c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:15.668697 8220 tablet_bootstrap.cc:492] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c: Bootstrap starting.
I20240502 01:54:15.670833 8220 tablet_bootstrap.cc:654] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.672253 8226 tablet_bootstrap.cc:492] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:15.672530 8226 ts_tablet_manager.cc:1367] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240502 01:54:15.673430 8226 raft_consensus.cc:348] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:15.673692 8226 raft_consensus.cc:374] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.673810 8226 raft_consensus.cc:729] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.674126 8226 consensus_queue.cc:260] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:15.675740 8226 ts_tablet_manager.cc:1398] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:15.679126 8220 tablet_bootstrap.cc:492] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c: No bootstrap required, opened a new log
I20240502 01:54:15.679301 8220 ts_tablet_manager.cc:1367] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240502 01:54:15.680370 8220 raft_consensus.cc:348] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:15.680604 8220 raft_consensus.cc:374] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.680678 8220 raft_consensus.cc:729] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.680945 8220 consensus_queue.cc:260] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:15.682091 8220 ts_tablet_manager.cc:1398] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:15.682413 8220 tablet_bootstrap.cc:492] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c: Bootstrap starting.
I20240502 01:54:15.684211 8220 tablet_bootstrap.cc:654] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.686305 8219 tablet_bootstrap.cc:492] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17: No bootstrap required, opened a new log
I20240502 01:54:15.686506 8219 ts_tablet_manager.cc:1367] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.010s
I20240502 01:54:15.687422 8219 raft_consensus.cc:348] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:15.687670 8219 raft_consensus.cc:374] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.687754 8219 raft_consensus.cc:729] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.688041 8219 consensus_queue.cc:260] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:15.688750 8219 ts_tablet_manager.cc:1398] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:15.689745 8220 tablet_bootstrap.cc:492] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c: No bootstrap required, opened a new log
I20240502 01:54:15.689886 8220 ts_tablet_manager.cc:1367] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 01:54:15.690601 8220 raft_consensus.cc:348] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:15.690810 8220 raft_consensus.cc:374] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.690889 8220 raft_consensus.cc:729] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.691092 8220 consensus_queue.cc:260] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:15.691707 8220 ts_tablet_manager.cc:1398] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 01:54:15.694329 8221 tablet_bootstrap.cc:492] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:15.694535 8221 ts_tablet_manager.cc:1367] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.059s user 0.010s sys 0.031s
I20240502 01:54:15.695370 8221 raft_consensus.cc:348] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:15.695555 8221 raft_consensus.cc:374] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.695644 8221 raft_consensus.cc:729] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.695924 8221 consensus_queue.cc:260] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:15.696435 8221 ts_tablet_manager.cc:1398] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:54:15.696745 8221 tablet_bootstrap.cc:492] T 482f77b65a254ed3aec717c06698ffa7 P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:15.698282 8221 tablet_bootstrap.cc:654] T 482f77b65a254ed3aec717c06698ffa7 P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.700037 8221 tablet_bootstrap.cc:492] T 482f77b65a254ed3aec717c06698ffa7 P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:15.700206 8221 ts_tablet_manager.cc:1367] T 482f77b65a254ed3aec717c06698ffa7 P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 01:54:15.700837 8221 raft_consensus.cc:348] T 482f77b65a254ed3aec717c06698ffa7 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:15.701005 8221 raft_consensus.cc:374] T 482f77b65a254ed3aec717c06698ffa7 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.701058 8221 raft_consensus.cc:729] T 482f77b65a254ed3aec717c06698ffa7 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.701297 8221 consensus_queue.cc:260] T 482f77b65a254ed3aec717c06698ffa7 P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:15.701709 8221 ts_tablet_manager.cc:1398] T 482f77b65a254ed3aec717c06698ffa7 P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 01:54:15.702057 8221 tablet_bootstrap.cc:492] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:15.703447 8221 tablet_bootstrap.cc:654] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:15.705231 8221 tablet_bootstrap.cc:492] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:15.705384 8221 ts_tablet_manager.cc:1367] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240502 01:54:15.705965 8221 raft_consensus.cc:348] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:15.706102 8221 raft_consensus.cc:374] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:15.706164 8221 raft_consensus.cc:729] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:15.706329 8221 consensus_queue.cc:260] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:15.706796 8221 ts_tablet_manager.cc:1398] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:54:15.797924 8161 raft_consensus.cc:482] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:15.798131 8161 raft_consensus.cc:504] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:15.798987 8161 leader_election.cc:288] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315), 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:15.799667 7067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f4c94a105741339ba69e641244ac18" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3" is_pre_election: true
I20240502 01:54:15.799674 6937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f4c94a105741339ba69e641244ac18" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c" is_pre_election: true
I20240502 01:54:15.799896 6937 raft_consensus.cc:2454] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 0.
I20240502 01:54:15.799937 7067 raft_consensus.cc:2454] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 0.
I20240502 01:54:15.800467 6738 leader_election.cc:302] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 8214b6c4de6d4aa58a22af551eb8dedd; no voters:
I20240502 01:54:15.800845 7959 raft_consensus.cc:2781] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:15.800944 7959 raft_consensus.cc:482] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:15.801048 7959 raft_consensus.cc:3037] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.803129 7959 raft_consensus.cc:504] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:15.803390 8225 raft_consensus.cc:482] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:15.803534 8225 raft_consensus.cc:504] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:15.803715 7959 leader_election.cc:288] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 election: Requested vote from peers 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315), 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:15.804216 7067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f4c94a105741339ba69e641244ac18" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
I20240502 01:54:15.804229 6937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f4c94a105741339ba69e641244ac18" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c"
I20240502 01:54:15.804289 8225 leader_election.cc:288] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:15.804422 6937 raft_consensus.cc:3037] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.804432 7067 raft_consensus.cc:3037] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.804725 6936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "482f77b65a254ed3aec717c06698ffa7" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c" is_pre_election: true
I20240502 01:54:15.804901 6936 raft_consensus.cc:2454] T 482f77b65a254ed3aec717c06698ffa7 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 0.
I20240502 01:54:15.805225 7134 leader_election.cc:302] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 137e2183ff57411791e9af58dfab0a3c, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:15.805444 8225 raft_consensus.cc:2781] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:15.805528 8225 raft_consensus.cc:482] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:15.805583 8225 raft_consensus.cc:3037] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.807130 7067 raft_consensus.cc:2454] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 1.
I20240502 01:54:15.807142 6937 raft_consensus.cc:2454] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 1.
I20240502 01:54:15.807415 8225 raft_consensus.cc:504] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:15.807541 6738 leader_election.cc:302] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 8214b6c4de6d4aa58a22af551eb8dedd; no voters:
I20240502 01:54:15.807664 7067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "482f77b65a254ed3aec717c06698ffa7" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3" is_pre_election: true
I20240502 01:54:15.807792 7959 raft_consensus.cc:2781] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:15.807898 7067 raft_consensus.cc:2454] T 482f77b65a254ed3aec717c06698ffa7 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 0.
I20240502 01:54:15.807981 7959 raft_consensus.cc:686] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Becoming Leader. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Running, Role: LEADER
I20240502 01:54:15.808205 7959 consensus_queue.cc:237] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:15.808351 7067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "482f77b65a254ed3aec717c06698ffa7" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
I20240502 01:54:15.808521 7067 raft_consensus.cc:3037] T 482f77b65a254ed3aec717c06698ffa7 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.809044 8225 leader_election.cc:288] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 election: Requested vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:15.810256 6937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "482f77b65a254ed3aec717c06698ffa7" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c"
I20240502 01:54:15.810460 6937 raft_consensus.cc:3037] T 482f77b65a254ed3aec717c06698ffa7 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:15.811139 7067 raft_consensus.cc:2454] T 482f77b65a254ed3aec717c06698ffa7 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 1.
I20240502 01:54:15.811239 6656 catalog_manager.cc:5484] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193). New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:15.811530 7134 leader_election.cc:302] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:15.812261 8225 raft_consensus.cc:2781] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:15.812417 8225 raft_consensus.cc:686] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Becoming Leader. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Running, Role: LEADER
I20240502 01:54:15.812489 6937 raft_consensus.cc:2454] T 482f77b65a254ed3aec717c06698ffa7 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 1.
I20240502 01:54:15.812716 8225 consensus_queue.cc:237] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:15.815411 6656 catalog_manager.cc:5484] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc21155d36d5416fa3650f0268a20b17 (127.31.161.196). New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:16.049650 8230 raft_consensus.cc:482] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:16.049870 8230 raft_consensus.cc:504] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:16.050575 8230 leader_election.cc:288] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873)
I20240502 01:54:16.051218 7066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fb5e1f69284140a6c6da264a124b3e" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3" is_pre_election: true
I20240502 01:54:16.051218 7198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fb5e1f69284140a6c6da264a124b3e" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17" is_pre_election: true
I20240502 01:54:16.051486 7198 raft_consensus.cc:2454] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 0.
I20240502 01:54:16.051472 7066 raft_consensus.cc:2454] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 0.
I20240502 01:54:16.051954 6870 leader_election.cc:302] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 137e2183ff57411791e9af58dfab0a3c, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:16.052237 8230 raft_consensus.cc:2781] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:16.052351 8230 raft_consensus.cc:482] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:16.052445 8230 raft_consensus.cc:3037] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.054373 8230 raft_consensus.cc:504] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:16.054905 8230 leader_election.cc:288] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 election: Requested vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873)
I20240502 01:54:16.055284 7067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fb5e1f69284140a6c6da264a124b3e" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
I20240502 01:54:16.055374 7198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fb5e1f69284140a6c6da264a124b3e" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17"
I20240502 01:54:16.055531 7067 raft_consensus.cc:3037] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.055575 7198 raft_consensus.cc:3037] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:16.058210 7198 raft_consensus.cc:2454] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 1.
I20240502 01:54:16.058209 7067 raft_consensus.cc:2454] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 1.
I20240502 01:54:16.058605 6871 leader_election.cc:302] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 137e2183ff57411791e9af58dfab0a3c; no voters:
I20240502 01:54:16.058892 8230 raft_consensus.cc:2781] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:16.059020 8230 raft_consensus.cc:686] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Becoming Leader. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Running, Role: LEADER
I20240502 01:54:16.059281 8230 consensus_queue.cc:237] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:16.062086 6656 catalog_manager.cc:5484] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 137e2183ff57411791e9af58dfab0a3c (127.31.161.194). New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:16.205432 8225 consensus_queue.cc:1035] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.211045 8240 consensus_queue.cc:1035] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.254427 8161 consensus_queue.cc:1035] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.262665 8246 consensus_queue.cc:1035] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:16.438391 8237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:16.438694 8237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:16.445955 8237 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 01:54:16.581293 7198 raft_consensus.cc:1261] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:16.586035 8230 consensus_queue.cc:1035] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.587486 7067 raft_consensus.cc:1261] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:54:16.595539 8230 consensus_queue.cc:1035] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:16.634987 8279 mvcc.cc:205] Tried to move back new op lower bound from 7023062452551221248 to 7023062450420785152. Current Snapshot: MvccSnapshot[applied={T|T < 7023062452551221248}]
INSERT report
rows total: 2000
time total: 239.415 ms
time per row: 0.119707 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 01:54:17.707792 6658 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43140:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:17.717880 7179 tablet_service.cc:1467] Processing CreateTablet for tablet db7a48f131f141f0a4e45036434f8129 (DEFAULT_TABLE table=rebalance_test_table_6 [id=981b993f808c4320845154760fb7959f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:17.718176 6784 tablet_service.cc:1467] Processing CreateTablet for tablet db7a48f131f141f0a4e45036434f8129 (DEFAULT_TABLE table=rebalance_test_table_6 [id=981b993f808c4320845154760fb7959f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:17.718374 7179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db7a48f131f141f0a4e45036434f8129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:17.718603 6784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db7a48f131f141f0a4e45036434f8129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:17.718580 6915 tablet_service.cc:1467] Processing CreateTablet for tablet db7a48f131f141f0a4e45036434f8129 (DEFAULT_TABLE table=rebalance_test_table_6 [id=981b993f808c4320845154760fb7959f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:17.718977 6915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db7a48f131f141f0a4e45036434f8129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:17.720542 6916 tablet_service.cc:1467] Processing CreateTablet for tablet 8960ca8e5d26429589abc4a154311983 (DEFAULT_TABLE table=rebalance_test_table_6 [id=981b993f808c4320845154760fb7959f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:17.720930 6916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8960ca8e5d26429589abc4a154311983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:17.721485 7047 tablet_service.cc:1467] Processing CreateTablet for tablet dabfafc485c144b995aa4248fb1e897b (DEFAULT_TABLE table=rebalance_test_table_6 [id=981b993f808c4320845154760fb7959f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:17.721864 7047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dabfafc485c144b995aa4248fb1e897b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:17.725600 7177 tablet_service.cc:1467] Processing CreateTablet for tablet dabfafc485c144b995aa4248fb1e897b (DEFAULT_TABLE table=rebalance_test_table_6 [id=981b993f808c4320845154760fb7959f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:17.725673 6783 tablet_service.cc:1467] Processing CreateTablet for tablet dabfafc485c144b995aa4248fb1e897b (DEFAULT_TABLE table=rebalance_test_table_6 [id=981b993f808c4320845154760fb7959f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:17.725967 7177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dabfafc485c144b995aa4248fb1e897b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:17.726137 6783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dabfafc485c144b995aa4248fb1e897b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:17.725677 6782 tablet_service.cc:1467] Processing CreateTablet for tablet 8960ca8e5d26429589abc4a154311983 (DEFAULT_TABLE table=rebalance_test_table_6 [id=981b993f808c4320845154760fb7959f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:17.729498 6782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8960ca8e5d26429589abc4a154311983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:17.729419 7178 tablet_service.cc:1467] Processing CreateTablet for tablet 8960ca8e5d26429589abc4a154311983 (DEFAULT_TABLE table=rebalance_test_table_6 [id=981b993f808c4320845154760fb7959f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:17.729869 7178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8960ca8e5d26429589abc4a154311983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:17.739624 8303 tablet_bootstrap.cc:492] T db7a48f131f141f0a4e45036434f8129 P fc21155d36d5416fa3650f0268a20b17: Bootstrap starting.
I20240502 01:54:17.741755 8306 tablet_bootstrap.cc:492] T dabfafc485c144b995aa4248fb1e897b P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:17.743464 8304 tablet_bootstrap.cc:492] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:17.743669 8306 tablet_bootstrap.cc:654] T dabfafc485c144b995aa4248fb1e897b P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.743687 8303 tablet_bootstrap.cc:654] T db7a48f131f141f0a4e45036434f8129 P fc21155d36d5416fa3650f0268a20b17: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.746217 8304 tablet_bootstrap.cc:654] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.747277 8305 tablet_bootstrap.cc:492] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c: Bootstrap starting.
I20240502 01:54:17.748994 8305 tablet_bootstrap.cc:654] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.754300 8303 tablet_bootstrap.cc:492] T db7a48f131f141f0a4e45036434f8129 P fc21155d36d5416fa3650f0268a20b17: No bootstrap required, opened a new log
I20240502 01:54:17.754525 8303 ts_tablet_manager.cc:1367] T db7a48f131f141f0a4e45036434f8129 P fc21155d36d5416fa3650f0268a20b17: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20240502 01:54:17.755375 8303 raft_consensus.cc:348] T db7a48f131f141f0a4e45036434f8129 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:17.755735 8303 raft_consensus.cc:374] T db7a48f131f141f0a4e45036434f8129 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.758196 8303 raft_consensus.cc:729] T db7a48f131f141f0a4e45036434f8129 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.758602 8303 consensus_queue.cc:260] T db7a48f131f141f0a4e45036434f8129 P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:17.759416 8303 ts_tablet_manager.cc:1398] T db7a48f131f141f0a4e45036434f8129 P fc21155d36d5416fa3650f0268a20b17: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:54:17.761205 8305 tablet_bootstrap.cc:492] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c: No bootstrap required, opened a new log
I20240502 01:54:17.761445 8305 ts_tablet_manager.cc:1367] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.004s
I20240502 01:54:17.761655 8303 tablet_bootstrap.cc:492] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17: Bootstrap starting.
I20240502 01:54:17.763067 8305 raft_consensus.cc:348] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:17.763628 8305 raft_consensus.cc:374] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.763782 8305 raft_consensus.cc:729] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.764979 8304 tablet_bootstrap.cc:492] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:17.765211 8304 ts_tablet_manager.cc:1367] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.006s
I20240502 01:54:17.766017 8304 raft_consensus.cc:348] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:17.766091 8306 tablet_bootstrap.cc:492] T dabfafc485c144b995aa4248fb1e897b P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:17.766289 8304 raft_consensus.cc:374] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.766398 8306 ts_tablet_manager.cc:1367] T dabfafc485c144b995aa4248fb1e897b P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.008s
I20240502 01:54:17.766453 8304 raft_consensus.cc:729] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.766764 8303 tablet_bootstrap.cc:654] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.766791 8304 consensus_queue.cc:260] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:17.766885 8305 consensus_queue.cc:260] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:17.767721 8304 ts_tablet_manager.cc:1398] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:17.767266 8306 raft_consensus.cc:348] T dabfafc485c144b995aa4248fb1e897b P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:17.768245 8304 tablet_bootstrap.cc:492] T dabfafc485c144b995aa4248fb1e897b P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:17.768229 8306 raft_consensus.cc:374] T dabfafc485c144b995aa4248fb1e897b P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.768333 8306 raft_consensus.cc:729] T dabfafc485c144b995aa4248fb1e897b P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.768388 8305 ts_tablet_manager.cc:1398] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 01:54:17.768687 8305 tablet_bootstrap.cc:492] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c: Bootstrap starting.
I20240502 01:54:17.768677 8306 consensus_queue.cc:260] T dabfafc485c144b995aa4248fb1e897b P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:17.769852 8304 tablet_bootstrap.cc:654] T dabfafc485c144b995aa4248fb1e897b P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.770170 8306 ts_tablet_manager.cc:1398] T dabfafc485c144b995aa4248fb1e897b P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:54:17.771044 8305 tablet_bootstrap.cc:654] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.773558 8304 tablet_bootstrap.cc:492] T dabfafc485c144b995aa4248fb1e897b P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:17.773752 8304 ts_tablet_manager.cc:1367] T dabfafc485c144b995aa4248fb1e897b P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240502 01:54:17.774502 8304 raft_consensus.cc:348] T dabfafc485c144b995aa4248fb1e897b P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:17.774866 8304 raft_consensus.cc:374] T dabfafc485c144b995aa4248fb1e897b P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.774996 8304 raft_consensus.cc:729] T dabfafc485c144b995aa4248fb1e897b P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.775482 8304 consensus_queue.cc:260] T dabfafc485c144b995aa4248fb1e897b P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:17.776269 8304 ts_tablet_manager.cc:1398] T dabfafc485c144b995aa4248fb1e897b P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:17.778357 8304 tablet_bootstrap.cc:492] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:17.781682 8304 tablet_bootstrap.cc:654] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.782065 8305 tablet_bootstrap.cc:492] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c: No bootstrap required, opened a new log
I20240502 01:54:17.782104 8303 tablet_bootstrap.cc:492] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17: No bootstrap required, opened a new log
I20240502 01:54:17.782254 8303 ts_tablet_manager.cc:1367] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.004s
I20240502 01:54:17.782254 8305 ts_tablet_manager.cc:1367] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.002s
I20240502 01:54:17.782984 8303 raft_consensus.cc:348] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:17.782984 8305 raft_consensus.cc:348] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:17.783128 8304 tablet_bootstrap.cc:492] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:17.783210 8305 raft_consensus.cc:374] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.783236 8304 ts_tablet_manager.cc:1367] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:54:17.783241 8303 raft_consensus.cc:374] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.783298 8305 raft_consensus.cc:729] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.783308 8303 raft_consensus.cc:729] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.783540 8305 consensus_queue.cc:260] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:17.783533 8303 consensus_queue.cc:260] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:17.784088 8305 ts_tablet_manager.cc:1398] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:17.784605 8303 ts_tablet_manager.cc:1398] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:17.784847 8303 tablet_bootstrap.cc:492] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17: Bootstrap starting.
I20240502 01:54:17.784574 8304 raft_consensus.cc:348] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:17.785952 8304 raft_consensus.cc:374] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.786072 8304 raft_consensus.cc:729] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.786104 8303 tablet_bootstrap.cc:654] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:17.786294 8304 consensus_queue.cc:260] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:17.786906 8304 ts_tablet_manager.cc:1398] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:54:17.790618 8303 tablet_bootstrap.cc:492] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17: No bootstrap required, opened a new log
I20240502 01:54:17.790748 8303 ts_tablet_manager.cc:1367] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240502 01:54:17.791290 8303 raft_consensus.cc:348] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:17.791437 8303 raft_consensus.cc:374] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:17.791563 8303 raft_consensus.cc:729] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Initialized, Role: FOLLOWER
I20240502 01:54:17.791819 8303 consensus_queue.cc:260] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:17.792313 8303 ts_tablet_manager.cc:1398] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 01:54:17.801121 8240 raft_consensus.cc:482] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:17.801265 8240 raft_consensus.cc:504] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:17.801759 8240 leader_election.cc:288] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:17.802192 6803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dabfafc485c144b995aa4248fb1e897b" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" is_pre_election: true
I20240502 01:54:17.802273 7066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dabfafc485c144b995aa4248fb1e897b" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3" is_pre_election: true
I20240502 01:54:17.802443 6803 raft_consensus.cc:2454] T dabfafc485c144b995aa4248fb1e897b P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 0.
I20240502 01:54:17.802495 7066 raft_consensus.cc:2454] T dabfafc485c144b995aa4248fb1e897b P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 0.
I20240502 01:54:17.802877 7133 leader_election.cc:302] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8214b6c4de6d4aa58a22af551eb8dedd, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:17.803203 8240 raft_consensus.cc:2781] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:17.803303 8240 raft_consensus.cc:482] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:17.803418 8240 raft_consensus.cc:3037] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.805747 8240 raft_consensus.cc:504] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:17.806416 8240 leader_election.cc:288] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 election: Requested vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:17.806946 6803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dabfafc485c144b995aa4248fb1e897b" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
I20240502 01:54:17.807013 7066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dabfafc485c144b995aa4248fb1e897b" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
I20240502 01:54:17.807178 6803 raft_consensus.cc:3037] T dabfafc485c144b995aa4248fb1e897b P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.807204 7066 raft_consensus.cc:3037] T dabfafc485c144b995aa4248fb1e897b P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.809419 6803 raft_consensus.cc:2454] T dabfafc485c144b995aa4248fb1e897b P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 1.
I20240502 01:54:17.809514 7066 raft_consensus.cc:2454] T dabfafc485c144b995aa4248fb1e897b P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 1.
I20240502 01:54:17.809770 7133 leader_election.cc:302] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8214b6c4de6d4aa58a22af551eb8dedd, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:17.809895 8278 raft_consensus.cc:482] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:17.810052 8278 raft_consensus.cc:504] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:17.810604 8278 leader_election.cc:288] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873), 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:17.810791 8240 raft_consensus.cc:2781] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:17.810972 8240 raft_consensus.cc:686] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Becoming Leader. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Running, Role: LEADER
I20240502 01:54:17.811046 7198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7a48f131f141f0a4e45036434f8129" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17" is_pre_election: true
I20240502 01:54:17.811195 6803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7a48f131f141f0a4e45036434f8129" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" is_pre_election: true
I20240502 01:54:17.811250 8240 consensus_queue.cc:237] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:17.811378 6803 raft_consensus.cc:2454] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 0.
I20240502 01:54:17.811348 7198 raft_consensus.cc:2454] T db7a48f131f141f0a4e45036434f8129 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 0.
I20240502 01:54:17.811825 6870 leader_election.cc:302] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 137e2183ff57411791e9af58dfab0a3c, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:17.812099 8278 raft_consensus.cc:2781] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:17.812230 8278 raft_consensus.cc:482] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:17.812315 8278 raft_consensus.cc:3037] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.814514 8278 raft_consensus.cc:504] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:17.815131 8278 leader_election.cc:288] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 election: Requested vote from peers fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873), 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:17.815529 7198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7a48f131f141f0a4e45036434f8129" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17"
I20240502 01:54:17.815378 6653 catalog_manager.cc:5484] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc21155d36d5416fa3650f0268a20b17 (127.31.161.196). New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:17.815690 7198 raft_consensus.cc:3037] T db7a48f131f141f0a4e45036434f8129 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.815693 6803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db7a48f131f141f0a4e45036434f8129" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
I20240502 01:54:17.815910 6803 raft_consensus.cc:3037] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.817917 7198 raft_consensus.cc:2454] T db7a48f131f141f0a4e45036434f8129 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 1.
I20240502 01:54:17.818084 6803 raft_consensus.cc:2454] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 1.
I20240502 01:54:17.818282 6870 leader_election.cc:302] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 137e2183ff57411791e9af58dfab0a3c, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:17.818622 8278 raft_consensus.cc:2781] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:17.819087 8278 raft_consensus.cc:686] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Becoming Leader. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Running, Role: LEADER
I20240502 01:54:17.819583 8278 consensus_queue.cc:237] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:17.822069 6653 catalog_manager.cc:5484] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 137e2183ff57411791e9af58dfab0a3c (127.31.161.194). New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:17.898046 8278 raft_consensus.cc:482] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:17.898288 8278 raft_consensus.cc:504] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:17.898895 8278 leader_election.cc:288] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873)
I20240502 01:54:17.899458 6803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8960ca8e5d26429589abc4a154311983" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" is_pre_election: true
I20240502 01:54:17.899485 7198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8960ca8e5d26429589abc4a154311983" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17" is_pre_election: true
I20240502 01:54:17.899724 7198 raft_consensus.cc:2454] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 0.
I20240502 01:54:17.899742 6803 raft_consensus.cc:2454] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 0.
I20240502 01:54:17.900205 6870 leader_election.cc:302] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 137e2183ff57411791e9af58dfab0a3c, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:17.900609 8278 raft_consensus.cc:2781] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:17.900717 8278 raft_consensus.cc:482] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:17.900826 8278 raft_consensus.cc:3037] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.902805 8278 raft_consensus.cc:504] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:17.903360 8278 leader_election.cc:288] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 election: Requested vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873)
I20240502 01:54:17.903697 6803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8960ca8e5d26429589abc4a154311983" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
I20240502 01:54:17.903888 6803 raft_consensus.cc:3037] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.903839 7198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8960ca8e5d26429589abc4a154311983" candidate_uuid: "137e2183ff57411791e9af58dfab0a3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17"
I20240502 01:54:17.904031 7198 raft_consensus.cc:3037] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:17.905727 6803 raft_consensus.cc:2454] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 1.
I20240502 01:54:17.906066 6870 leader_election.cc:302] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 137e2183ff57411791e9af58dfab0a3c, 8214b6c4de6d4aa58a22af551eb8dedd; no voters:
I20240502 01:54:17.906143 7198 raft_consensus.cc:2454] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 137e2183ff57411791e9af58dfab0a3c in term 1.
I20240502 01:54:17.906337 8278 raft_consensus.cc:2781] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:17.906517 8278 raft_consensus.cc:686] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Becoming Leader. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Running, Role: LEADER
I20240502 01:54:17.906769 8278 consensus_queue.cc:237] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:17.909723 6653 catalog_manager.cc:5484] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 137e2183ff57411791e9af58dfab0a3c (127.31.161.194). New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:18.193915 8240 consensus_queue.cc:1035] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:18.200620 8240 consensus_queue.cc:1035] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:18.266018 8320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:18.266306 8320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:18.273761 8320 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 01:54:18.310163 8278 consensus_queue.cc:1035] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:18.315536 8317 consensus_queue.cc:1035] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:18.319114 8345 consensus_queue.cc:1035] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 01:54:18.328145 8317 consensus_queue.cc:1035] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 206.678 ms
time per row: 0.103339 ms
I20240502 01:54:18.645787 6653 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43140:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:18.656049 6784 tablet_service.cc:1467] Processing CreateTablet for tablet 43e6e5b3435a446499106efe7baf92aa (DEFAULT_TABLE table=rebalance_test_table_7 [id=4d5b7e86d6b24299a377cfaff5756f47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.656095 7047 tablet_service.cc:1467] Processing CreateTablet for tablet 43e6e5b3435a446499106efe7baf92aa (DEFAULT_TABLE table=rebalance_test_table_7 [id=4d5b7e86d6b24299a377cfaff5756f47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.656528 6784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e6e5b3435a446499106efe7baf92aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.656556 7047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e6e5b3435a446499106efe7baf92aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.656512 7179 tablet_service.cc:1467] Processing CreateTablet for tablet ae02d485bc56444e994a56f759fbed1f (DEFAULT_TABLE table=rebalance_test_table_7 [id=4d5b7e86d6b24299a377cfaff5756f47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.656924 7179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae02d485bc56444e994a56f759fbed1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.661933 7048 tablet_service.cc:1467] Processing CreateTablet for tablet ae02d485bc56444e994a56f759fbed1f (DEFAULT_TABLE table=rebalance_test_table_7 [id=4d5b7e86d6b24299a377cfaff5756f47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.661935 6915 tablet_service.cc:1467] Processing CreateTablet for tablet 43e6e5b3435a446499106efe7baf92aa (DEFAULT_TABLE table=rebalance_test_table_7 [id=4d5b7e86d6b24299a377cfaff5756f47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.662318 7048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae02d485bc56444e994a56f759fbed1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.662396 6915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e6e5b3435a446499106efe7baf92aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.666116 6916 tablet_service.cc:1467] Processing CreateTablet for tablet ae02d485bc56444e994a56f759fbed1f (DEFAULT_TABLE table=rebalance_test_table_7 [id=4d5b7e86d6b24299a377cfaff5756f47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.666508 6916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae02d485bc56444e994a56f759fbed1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.666420 6783 tablet_service.cc:1467] Processing CreateTablet for tablet d58f60c0920e4616850cb246acce1fd7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4d5b7e86d6b24299a377cfaff5756f47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.666796 6783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58f60c0920e4616850cb246acce1fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.666813 7046 tablet_service.cc:1467] Processing CreateTablet for tablet d58f60c0920e4616850cb246acce1fd7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4d5b7e86d6b24299a377cfaff5756f47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.667187 7046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58f60c0920e4616850cb246acce1fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.669927 7178 tablet_service.cc:1467] Processing CreateTablet for tablet d58f60c0920e4616850cb246acce1fd7 (DEFAULT_TABLE table=rebalance_test_table_7 [id=4d5b7e86d6b24299a377cfaff5756f47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:18.674949 8385 tablet_bootstrap.cc:492] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:18.676525 8386 tablet_bootstrap.cc:492] T ae02d485bc56444e994a56f759fbed1f P 137e2183ff57411791e9af58dfab0a3c: Bootstrap starting.
I20240502 01:54:18.676728 8385 tablet_bootstrap.cc:654] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.677712 7178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58f60c0920e4616850cb246acce1fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:18.679622 8386 tablet_bootstrap.cc:654] T ae02d485bc56444e994a56f759fbed1f P 137e2183ff57411791e9af58dfab0a3c: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.681020 8387 tablet_bootstrap.cc:492] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17: Bootstrap starting.
I20240502 01:54:18.684568 8387 tablet_bootstrap.cc:654] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.685624 8384 tablet_bootstrap.cc:492] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:18.686643 8385 tablet_bootstrap.cc:492] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:18.686827 8385 ts_tablet_manager.cc:1367] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20240502 01:54:18.687429 8384 tablet_bootstrap.cc:654] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.687460 8386 tablet_bootstrap.cc:492] T ae02d485bc56444e994a56f759fbed1f P 137e2183ff57411791e9af58dfab0a3c: No bootstrap required, opened a new log
I20240502 01:54:18.687594 8386 ts_tablet_manager.cc:1367] T ae02d485bc56444e994a56f759fbed1f P 137e2183ff57411791e9af58dfab0a3c: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240502 01:54:18.687649 8385 raft_consensus.cc:348] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:18.687858 8385 raft_consensus.cc:374] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.687964 8385 raft_consensus.cc:729] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.688198 8385 consensus_queue.cc:260] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:18.688756 8385 ts_tablet_manager.cc:1398] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:18.689049 8385 tablet_bootstrap.cc:492] T ae02d485bc56444e994a56f759fbed1f P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:18.690194 8384 tablet_bootstrap.cc:492] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:18.690375 8384 ts_tablet_manager.cc:1367] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:54:18.690577 8385 tablet_bootstrap.cc:654] T ae02d485bc56444e994a56f759fbed1f P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.691180 8384 raft_consensus.cc:348] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:18.692442 8384 raft_consensus.cc:374] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.692566 8384 raft_consensus.cc:729] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.692833 8384 consensus_queue.cc:260] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:18.693437 8384 ts_tablet_manager.cc:1398] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 01:54:18.696121 8386 raft_consensus.cc:348] T ae02d485bc56444e994a56f759fbed1f P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:18.696389 8386 raft_consensus.cc:374] T ae02d485bc56444e994a56f759fbed1f P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.696487 8386 raft_consensus.cc:729] T ae02d485bc56444e994a56f759fbed1f P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.696764 8386 consensus_queue.cc:260] T ae02d485bc56444e994a56f759fbed1f P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:18.698310 8386 ts_tablet_manager.cc:1398] T ae02d485bc56444e994a56f759fbed1f P 137e2183ff57411791e9af58dfab0a3c: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:54:18.698617 8386 tablet_bootstrap.cc:492] T 43e6e5b3435a446499106efe7baf92aa P 137e2183ff57411791e9af58dfab0a3c: Bootstrap starting.
I20240502 01:54:18.701009 8386 tablet_bootstrap.cc:654] T 43e6e5b3435a446499106efe7baf92aa P 137e2183ff57411791e9af58dfab0a3c: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.702776 8384 tablet_bootstrap.cc:492] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:18.704718 8384 tablet_bootstrap.cc:654] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.710700 8387 tablet_bootstrap.cc:492] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17: No bootstrap required, opened a new log
I20240502 01:54:18.710744 8385 tablet_bootstrap.cc:492] T ae02d485bc56444e994a56f759fbed1f P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:18.710850 8384 tablet_bootstrap.cc:492] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:18.710925 8387 ts_tablet_manager.cc:1367] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.018s
I20240502 01:54:18.711045 8384 ts_tablet_manager.cc:1367] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240502 01:54:18.711040 8385 ts_tablet_manager.cc:1367] T ae02d485bc56444e994a56f759fbed1f P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.012s
I20240502 01:54:18.711812 8387 raft_consensus.cc:348] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:18.711795 8384 raft_consensus.cc:348] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:18.711823 8385 raft_consensus.cc:348] T ae02d485bc56444e994a56f759fbed1f P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:18.712024 8384 raft_consensus.cc:374] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.712102 8385 raft_consensus.cc:374] T ae02d485bc56444e994a56f759fbed1f P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.712119 8387 raft_consensus.cc:374] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.712127 8384 raft_consensus.cc:729] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.712245 8387 raft_consensus.cc:729] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.712327 8385 raft_consensus.cc:729] T ae02d485bc56444e994a56f759fbed1f P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.712378 8384 consensus_queue.cc:260] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:18.712565 8387 consensus_queue.cc:260] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:18.713095 8384 ts_tablet_manager.cc:1398] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:54:18.713274 8387 ts_tablet_manager.cc:1398] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:18.713531 8387 tablet_bootstrap.cc:492] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17: Bootstrap starting.
I20240502 01:54:18.712700 8385 consensus_queue.cc:260] T ae02d485bc56444e994a56f759fbed1f P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:18.714569 8385 ts_tablet_manager.cc:1398] T ae02d485bc56444e994a56f759fbed1f P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:54:18.714915 8385 tablet_bootstrap.cc:492] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:18.715003 8387 tablet_bootstrap.cc:654] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.716499 8385 tablet_bootstrap.cc:654] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:18.721594 8386 tablet_bootstrap.cc:492] T 43e6e5b3435a446499106efe7baf92aa P 137e2183ff57411791e9af58dfab0a3c: No bootstrap required, opened a new log
I20240502 01:54:18.721901 8386 ts_tablet_manager.cc:1367] T 43e6e5b3435a446499106efe7baf92aa P 137e2183ff57411791e9af58dfab0a3c: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20240502 01:54:18.722811 8386 raft_consensus.cc:348] T 43e6e5b3435a446499106efe7baf92aa P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:18.723106 8386 raft_consensus.cc:374] T 43e6e5b3435a446499106efe7baf92aa P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.723093 8298 raft_consensus.cc:482] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:18.723219 8386 raft_consensus.cc:729] T 43e6e5b3435a446499106efe7baf92aa P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.723254 8298 raft_consensus.cc:504] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:18.723511 8386 consensus_queue.cc:260] T 43e6e5b3435a446499106efe7baf92aa P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:18.724076 8387 tablet_bootstrap.cc:492] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17: No bootstrap required, opened a new log
I20240502 01:54:18.724077 8386 ts_tablet_manager.cc:1398] T 43e6e5b3435a446499106efe7baf92aa P 137e2183ff57411791e9af58dfab0a3c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:18.724103 8298 leader_election.cc:288] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:18.724247 8387 ts_tablet_manager.cc:1367] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20240502 01:54:18.724843 8387 raft_consensus.cc:348] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:18.725052 8387 raft_consensus.cc:374] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.725162 8387 raft_consensus.cc:729] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.725451 8387 consensus_queue.cc:260] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:18.725471 7066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae02d485bc56444e994a56f759fbed1f" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3" is_pre_election: true
I20240502 01:54:18.725745 7066 raft_consensus.cc:2454] T ae02d485bc56444e994a56f759fbed1f P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 0.
I20240502 01:54:18.726087 6935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae02d485bc56444e994a56f759fbed1f" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c" is_pre_election: true
I20240502 01:54:18.726179 7134 leader_election.cc:302] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:18.726271 6935 raft_consensus.cc:2454] T ae02d485bc56444e994a56f759fbed1f P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 0.
I20240502 01:54:18.726459 8298 raft_consensus.cc:2781] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:18.726584 8298 raft_consensus.cc:482] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:18.726686 8298 raft_consensus.cc:3037] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.726841 8387 ts_tablet_manager.cc:1398] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:18.727329 8385 tablet_bootstrap.cc:492] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:18.727494 8385 ts_tablet_manager.cc:1367] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20240502 01:54:18.728246 8385 raft_consensus.cc:348] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:18.728415 8385 raft_consensus.cc:374] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:18.728535 8385 raft_consensus.cc:729] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:18.728773 8385 consensus_queue.cc:260] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:18.729446 8298 raft_consensus.cc:504] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:18.730029 8385 ts_tablet_manager.cc:1398] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:18.730369 7066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae02d485bc56444e994a56f759fbed1f" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
I20240502 01:54:18.730543 8298 leader_election.cc:288] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 election: Requested vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:18.730561 7066 raft_consensus.cc:3037] T ae02d485bc56444e994a56f759fbed1f P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.731113 6935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae02d485bc56444e994a56f759fbed1f" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c"
I20240502 01:54:18.731285 6935 raft_consensus.cc:3037] T ae02d485bc56444e994a56f759fbed1f P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.733258 7066 raft_consensus.cc:2454] T ae02d485bc56444e994a56f759fbed1f P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 1.
I20240502 01:54:18.733892 6935 raft_consensus.cc:2454] T ae02d485bc56444e994a56f759fbed1f P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 1.
I20240502 01:54:18.733892 7134 leader_election.cc:302] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:18.734314 8298 raft_consensus.cc:2781] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:18.734500 8298 raft_consensus.cc:686] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Becoming Leader. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Running, Role: LEADER
I20240502 01:54:18.734817 8298 consensus_queue.cc:237] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:18.737491 6659 catalog_manager.cc:5484] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc21155d36d5416fa3650f0268a20b17 (127.31.161.196). New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:18.749609 8298 raft_consensus.cc:482] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:18.749840 8298 raft_consensus.cc:504] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:18.750921 8298 leader_election.cc:288] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:18.750928 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58f60c0920e4616850cb246acce1fd7" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" is_pre_election: true
I20240502 01:54:18.751050 7066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58f60c0920e4616850cb246acce1fd7" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3" is_pre_election: true
I20240502 01:54:18.751147 6804 raft_consensus.cc:2454] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 0.
I20240502 01:54:18.751267 7066 raft_consensus.cc:2454] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 0.
I20240502 01:54:18.751574 7133 leader_election.cc:302] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8214b6c4de6d4aa58a22af551eb8dedd, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:18.751888 8298 raft_consensus.cc:2781] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:18.751973 8298 raft_consensus.cc:482] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:18.752071 8298 raft_consensus.cc:3037] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.753901 8298 raft_consensus.cc:504] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:18.754714 8298 leader_election.cc:288] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 election: Requested vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:18.754866 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58f60c0920e4616850cb246acce1fd7" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
I20240502 01:54:18.755055 6804 raft_consensus.cc:3037] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.756948 6804 raft_consensus.cc:2454] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 1.
I20240502 01:54:18.757345 7133 leader_election.cc:302] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8214b6c4de6d4aa58a22af551eb8dedd, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:18.757457 7066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58f60c0920e4616850cb246acce1fd7" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
I20240502 01:54:18.757644 7066 raft_consensus.cc:3037] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.757668 8298 raft_consensus.cc:2781] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:18.758119 8298 raft_consensus.cc:686] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Becoming Leader. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Running, Role: LEADER
I20240502 01:54:18.758399 8298 consensus_queue.cc:237] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } }
I20240502 01:54:18.760020 7066 raft_consensus.cc:2454] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 1.
I20240502 01:54:18.761261 6659 catalog_manager.cc:5484] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc21155d36d5416fa3650f0268a20b17 (127.31.161.196). New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:18.768615 8158 raft_consensus.cc:482] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:18.768802 8158 raft_consensus.cc:504] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:18.769556 8158 leader_election.cc:288] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:18.770427 6935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e6e5b3435a446499106efe7baf92aa" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c" is_pre_election: true
I20240502 01:54:18.770694 6935 raft_consensus.cc:2454] T 43e6e5b3435a446499106efe7baf92aa P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 0.
I20240502 01:54:18.770782 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e6e5b3435a446499106efe7baf92aa" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" is_pre_election: true
I20240502 01:54:18.771028 6804 raft_consensus.cc:2454] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 0.
I20240502 01:54:18.771085 7003 leader_election.cc:302] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 137e2183ff57411791e9af58dfab0a3c; no voters:
I20240502 01:54:18.771340 8399 raft_consensus.cc:2781] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:18.771476 8399 raft_consensus.cc:482] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:18.771585 8399 raft_consensus.cc:3037] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.773527 8399 raft_consensus.cc:504] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:18.774075 8399 leader_election.cc:288] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 election: Requested vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:18.774462 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e6e5b3435a446499106efe7baf92aa" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
I20240502 01:54:18.774482 6935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e6e5b3435a446499106efe7baf92aa" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c"
I20240502 01:54:18.774645 6804 raft_consensus.cc:3037] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.774701 6935 raft_consensus.cc:3037] T 43e6e5b3435a446499106efe7baf92aa P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:18.776520 6804 raft_consensus.cc:2454] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 1.
I20240502 01:54:18.776604 6935 raft_consensus.cc:2454] T 43e6e5b3435a446499106efe7baf92aa P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 1.
I20240502 01:54:18.776985 7003 leader_election.cc:302] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 137e2183ff57411791e9af58dfab0a3c; no voters:
I20240502 01:54:18.777273 8399 raft_consensus.cc:2781] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:18.777420 8399 raft_consensus.cc:686] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Becoming Leader. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Running, Role: LEADER
I20240502 01:54:18.777740 8399 consensus_queue.cc:237] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:18.780452 6659 catalog_manager.cc:5484] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195). New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } }
W20240502 01:54:19.145795 8401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:19.146104 8401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:19.153789 8401 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:54:19.173308 8399 consensus_queue.cc:1035] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:19.180421 8158 consensus_queue.cc:1035] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:19.181843 8298 consensus_queue.cc:1035] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:19.188027 8298 consensus_queue.cc:1035] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
I20240502 01:54:19.283849 8240 consensus_queue.cc:1035] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:19.301180 8420 consensus_queue.cc:1035] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 208.595 ms
time per row: 0.104297 ms
I20240502 01:54:19.555111 6653 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43140:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:19.564049 7047 tablet_service.cc:1467] Processing CreateTablet for tablet c86cf318e8e34435bd1dac117d7cc389 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8abdf44bc9e2414b9e4f80bf4c87c953]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.564544 7047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86cf318e8e34435bd1dac117d7cc389. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.564489 6916 tablet_service.cc:1467] Processing CreateTablet for tablet c86cf318e8e34435bd1dac117d7cc389 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8abdf44bc9e2414b9e4f80bf4c87c953]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.564502 6784 tablet_service.cc:1467] Processing CreateTablet for tablet c86cf318e8e34435bd1dac117d7cc389 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8abdf44bc9e2414b9e4f80bf4c87c953]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.564843 6916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86cf318e8e34435bd1dac117d7cc389. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.564906 6784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86cf318e8e34435bd1dac117d7cc389. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.569901 6915 tablet_service.cc:1467] Processing CreateTablet for tablet 7d8c4a55e09d41e391b8ba318dfe6896 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8abdf44bc9e2414b9e4f80bf4c87c953]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.570341 6915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d8c4a55e09d41e391b8ba318dfe6896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.570789 6783 tablet_service.cc:1467] Processing CreateTablet for tablet 7d8c4a55e09d41e391b8ba318dfe6896 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8abdf44bc9e2414b9e4f80bf4c87c953]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.571169 6783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d8c4a55e09d41e391b8ba318dfe6896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.573923 7179 tablet_service.cc:1467] Processing CreateTablet for tablet 7d8c4a55e09d41e391b8ba318dfe6896 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8abdf44bc9e2414b9e4f80bf4c87c953]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.574357 7179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d8c4a55e09d41e391b8ba318dfe6896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.577952 7178 tablet_service.cc:1467] Processing CreateTablet for tablet 4b10e180c11c4cd4a308fc7d311840c2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8abdf44bc9e2414b9e4f80bf4c87c953]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.578351 7178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b10e180c11c4cd4a308fc7d311840c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.580226 8461 tablet_bootstrap.cc:492] T c86cf318e8e34435bd1dac117d7cc389 P 137e2183ff57411791e9af58dfab0a3c: Bootstrap starting.
I20240502 01:54:19.581970 8461 tablet_bootstrap.cc:654] T c86cf318e8e34435bd1dac117d7cc389 P 137e2183ff57411791e9af58dfab0a3c: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.583503 6916 tablet_service.cc:1467] Processing CreateTablet for tablet 4b10e180c11c4cd4a308fc7d311840c2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8abdf44bc9e2414b9e4f80bf4c87c953]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.583683 7046 tablet_service.cc:1467] Processing CreateTablet for tablet 4b10e180c11c4cd4a308fc7d311840c2 (DEFAULT_TABLE table=rebalance_test_table_8 [id=8abdf44bc9e2414b9e4f80bf4c87c953]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:19.583838 8462 tablet_bootstrap.cc:492] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:19.584071 6916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b10e180c11c4cd4a308fc7d311840c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.584085 7046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b10e180c11c4cd4a308fc7d311840c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:19.585722 8462 tablet_bootstrap.cc:654] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.587144 8463 tablet_bootstrap.cc:492] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:19.587991 8462 tablet_bootstrap.cc:492] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:19.588210 8462 ts_tablet_manager.cc:1367] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240502 01:54:19.588812 8463 tablet_bootstrap.cc:654] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.589779 8461 tablet_bootstrap.cc:492] T c86cf318e8e34435bd1dac117d7cc389 P 137e2183ff57411791e9af58dfab0a3c: No bootstrap required, opened a new log
I20240502 01:54:19.589123 8462 raft_consensus.cc:348] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:19.590533 8462 raft_consensus.cc:374] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.590649 8462 raft_consensus.cc:729] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.590838 8461 ts_tablet_manager.cc:1367] T c86cf318e8e34435bd1dac117d7cc389 P 137e2183ff57411791e9af58dfab0a3c: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20240502 01:54:19.590896 8462 consensus_queue.cc:260] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:19.591624 8462 ts_tablet_manager.cc:1398] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:19.591627 8461 raft_consensus.cc:348] T c86cf318e8e34435bd1dac117d7cc389 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:19.591835 8461 raft_consensus.cc:374] T c86cf318e8e34435bd1dac117d7cc389 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.591944 8461 raft_consensus.cc:729] T c86cf318e8e34435bd1dac117d7cc389 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.592181 8461 consensus_queue.cc:260] T c86cf318e8e34435bd1dac117d7cc389 P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:19.593488 8461 ts_tablet_manager.cc:1398] T c86cf318e8e34435bd1dac117d7cc389 P 137e2183ff57411791e9af58dfab0a3c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:19.593745 8461 tablet_bootstrap.cc:492] T 4b10e180c11c4cd4a308fc7d311840c2 P 137e2183ff57411791e9af58dfab0a3c: Bootstrap starting.
I20240502 01:54:19.595136 8461 tablet_bootstrap.cc:654] T 4b10e180c11c4cd4a308fc7d311840c2 P 137e2183ff57411791e9af58dfab0a3c: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.597878 8461 tablet_bootstrap.cc:492] T 4b10e180c11c4cd4a308fc7d311840c2 P 137e2183ff57411791e9af58dfab0a3c: No bootstrap required, opened a new log
I20240502 01:54:19.598032 8461 ts_tablet_manager.cc:1367] T 4b10e180c11c4cd4a308fc7d311840c2 P 137e2183ff57411791e9af58dfab0a3c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:54:19.598629 8461 raft_consensus.cc:348] T 4b10e180c11c4cd4a308fc7d311840c2 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:19.598815 8461 raft_consensus.cc:374] T 4b10e180c11c4cd4a308fc7d311840c2 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.598901 8461 raft_consensus.cc:729] T 4b10e180c11c4cd4a308fc7d311840c2 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.599124 8461 consensus_queue.cc:260] T 4b10e180c11c4cd4a308fc7d311840c2 P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:19.599521 8461 ts_tablet_manager.cc:1398] T 4b10e180c11c4cd4a308fc7d311840c2 P 137e2183ff57411791e9af58dfab0a3c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:54:19.599782 8461 tablet_bootstrap.cc:492] T 7d8c4a55e09d41e391b8ba318dfe6896 P 137e2183ff57411791e9af58dfab0a3c: Bootstrap starting.
I20240502 01:54:19.603607 8468 tablet_bootstrap.cc:492] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17: Bootstrap starting.
I20240502 01:54:19.603771 8462 tablet_bootstrap.cc:492] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:19.604085 8461 tablet_bootstrap.cc:654] T 7d8c4a55e09d41e391b8ba318dfe6896 P 137e2183ff57411791e9af58dfab0a3c: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.605333 8462 tablet_bootstrap.cc:654] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.606833 8468 tablet_bootstrap.cc:654] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.607121 8463 tablet_bootstrap.cc:492] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:19.607281 8463 ts_tablet_manager.cc:1367] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20240502 01:54:19.607980 8463 raft_consensus.cc:348] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:19.608219 8463 raft_consensus.cc:374] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.608335 8463 raft_consensus.cc:729] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.608568 8463 consensus_queue.cc:260] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:19.608979 8468 tablet_bootstrap.cc:492] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17: No bootstrap required, opened a new log
I20240502 01:54:19.609228 8468 ts_tablet_manager.cc:1367] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240502 01:54:19.609232 8463 ts_tablet_manager.cc:1398] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:19.609895 8461 tablet_bootstrap.cc:492] T 7d8c4a55e09d41e391b8ba318dfe6896 P 137e2183ff57411791e9af58dfab0a3c: No bootstrap required, opened a new log
I20240502 01:54:19.610035 8461 ts_tablet_manager.cc:1367] T 7d8c4a55e09d41e391b8ba318dfe6896 P 137e2183ff57411791e9af58dfab0a3c: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240502 01:54:19.610105 8468 raft_consensus.cc:348] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:19.610327 8468 raft_consensus.cc:374] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.610405 8468 raft_consensus.cc:729] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.610780 8461 raft_consensus.cc:348] T 7d8c4a55e09d41e391b8ba318dfe6896 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:19.610975 8461 raft_consensus.cc:374] T 7d8c4a55e09d41e391b8ba318dfe6896 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.611065 8461 raft_consensus.cc:729] T 7d8c4a55e09d41e391b8ba318dfe6896 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.611475 8463 tablet_bootstrap.cc:492] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:19.611793 8468 consensus_queue.cc:260] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:19.613106 8463 tablet_bootstrap.cc:654] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.613303 8461 consensus_queue.cc:260] T 7d8c4a55e09d41e391b8ba318dfe6896 P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:19.614172 8461 ts_tablet_manager.cc:1398] T 7d8c4a55e09d41e391b8ba318dfe6896 P 137e2183ff57411791e9af58dfab0a3c: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:54:19.623925 8468 ts_tablet_manager.cc:1398] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240502 01:54:19.625627 8468 tablet_bootstrap.cc:492] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17: Bootstrap starting.
I20240502 01:54:19.627508 8468 tablet_bootstrap.cc:654] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:19.630826 8462 tablet_bootstrap.cc:492] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:19.631017 8462 ts_tablet_manager.cc:1367] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.006s
I20240502 01:54:19.631938 8462 raft_consensus.cc:348] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:19.632198 8462 raft_consensus.cc:374] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.632269 8462 raft_consensus.cc:729] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.632521 8462 consensus_queue.cc:260] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:19.633287 8462 ts_tablet_manager.cc:1398] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 01:54:19.634992 8468 tablet_bootstrap.cc:492] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17: No bootstrap required, opened a new log
I20240502 01:54:19.635016 8463 tablet_bootstrap.cc:492] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:19.635156 8468 ts_tablet_manager.cc:1367] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240502 01:54:19.635197 8463 ts_tablet_manager.cc:1367] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.012s
I20240502 01:54:19.635802 8468 raft_consensus.cc:348] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:19.635996 8468 raft_consensus.cc:374] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.636075 8468 raft_consensus.cc:729] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.636016 8463 raft_consensus.cc:348] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:19.636240 8463 raft_consensus.cc:374] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:19.636319 8463 raft_consensus.cc:729] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:19.636349 8468 consensus_queue.cc:260] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:19.636582 8463 consensus_queue.cc:260] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:19.636914 8468 ts_tablet_manager.cc:1398] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:19.637122 8463 ts_tablet_manager.cc:1398] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 01:54:19.650255 8246 raft_consensus.cc:482] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:19.650465 8246 raft_consensus.cc:504] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:19.651139 8246 leader_election.cc:288] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:19.651626 7066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86cf318e8e34435bd1dac117d7cc389" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3" is_pre_election: true
I20240502 01:54:19.651767 6935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86cf318e8e34435bd1dac117d7cc389" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c" is_pre_election: true
I20240502 01:54:19.651892 7066 raft_consensus.cc:2454] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 0.
I20240502 01:54:19.652037 6935 raft_consensus.cc:2454] T c86cf318e8e34435bd1dac117d7cc389 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 0.
I20240502 01:54:19.652314 6738 leader_election.cc:302] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 8214b6c4de6d4aa58a22af551eb8dedd; no voters:
I20240502 01:54:19.652602 8246 raft_consensus.cc:2781] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:19.652710 8246 raft_consensus.cc:482] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:19.652804 8246 raft_consensus.cc:3037] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.655582 8246 raft_consensus.cc:504] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:19.656299 8246 leader_election.cc:288] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 election: Requested vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:19.656808 7066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86cf318e8e34435bd1dac117d7cc389" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
I20240502 01:54:19.656919 6935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c86cf318e8e34435bd1dac117d7cc389" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c"
I20240502 01:54:19.657042 7066 raft_consensus.cc:3037] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.657176 6935 raft_consensus.cc:3037] T c86cf318e8e34435bd1dac117d7cc389 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.660698 7066 raft_consensus.cc:2454] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 1.
I20240502 01:54:19.661031 6935 raft_consensus.cc:2454] T c86cf318e8e34435bd1dac117d7cc389 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 1.
I20240502 01:54:19.661170 6738 leader_election.cc:302] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 8214b6c4de6d4aa58a22af551eb8dedd; no voters:
I20240502 01:54:19.661492 8246 raft_consensus.cc:2781] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:19.661646 8246 raft_consensus.cc:686] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Becoming Leader. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Running, Role: LEADER
I20240502 01:54:19.661928 8246 consensus_queue.cc:237] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:19.665017 6652 catalog_manager.cc:5484] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193). New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:19.667776 8399 raft_consensus.cc:482] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:19.667946 8399 raft_consensus.cc:504] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:19.668565 8399 leader_election.cc:288] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:19.669099 6935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b10e180c11c4cd4a308fc7d311840c2" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c" is_pre_election: true
I20240502 01:54:19.669108 7197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b10e180c11c4cd4a308fc7d311840c2" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17" is_pre_election: true
I20240502 01:54:19.669332 7197 raft_consensus.cc:2454] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 0.
I20240502 01:54:19.669349 6935 raft_consensus.cc:2454] T 4b10e180c11c4cd4a308fc7d311840c2 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 0.
I20240502 01:54:19.669785 7003 leader_election.cc:302] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 137e2183ff57411791e9af58dfab0a3c; no voters:
I20240502 01:54:19.670116 8399 raft_consensus.cc:2781] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:19.670265 8399 raft_consensus.cc:482] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:19.670397 8399 raft_consensus.cc:3037] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.672394 8399 raft_consensus.cc:504] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:19.673025 8399 leader_election.cc:288] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 election: Requested vote from peers fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:19.673444 7197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b10e180c11c4cd4a308fc7d311840c2" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17"
I20240502 01:54:19.673511 6935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b10e180c11c4cd4a308fc7d311840c2" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c"
I20240502 01:54:19.673626 7197 raft_consensus.cc:3037] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.673720 6935 raft_consensus.cc:3037] T 4b10e180c11c4cd4a308fc7d311840c2 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.676179 6935 raft_consensus.cc:2454] T 4b10e180c11c4cd4a308fc7d311840c2 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 1.
I20240502 01:54:19.676201 7197 raft_consensus.cc:2454] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 1.
I20240502 01:54:19.676605 7003 leader_election.cc:302] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 137e2183ff57411791e9af58dfab0a3c; no voters:
I20240502 01:54:19.677021 8399 raft_consensus.cc:2781] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:19.677215 8399 raft_consensus.cc:686] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Becoming Leader. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Running, Role: LEADER
I20240502 01:54:19.677529 8399 consensus_queue.cc:237] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:19.680702 6652 catalog_manager.cc:5484] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195). New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:19.879751 8420 raft_consensus.cc:482] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:19.879992 8420 raft_consensus.cc:504] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:19.880621 8420 leader_election.cc:288] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315), 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:19.881114 6935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8c4a55e09d41e391b8ba318dfe6896" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c" is_pre_election: true
I20240502 01:54:19.881124 6802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8c4a55e09d41e391b8ba318dfe6896" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" is_pre_election: true
I20240502 01:54:19.881335 6935 raft_consensus.cc:2454] T 7d8c4a55e09d41e391b8ba318dfe6896 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 0.
I20240502 01:54:19.881335 6802 raft_consensus.cc:2454] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 0.
I20240502 01:54:19.881749 7134 leader_election.cc:302] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 137e2183ff57411791e9af58dfab0a3c, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:19.882025 8420 raft_consensus.cc:2781] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:19.882112 8420 raft_consensus.cc:482] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:19.882222 8420 raft_consensus.cc:3037] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.884079 8420 raft_consensus.cc:504] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:19.884605 8420 leader_election.cc:288] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 election: Requested vote from peers 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315), 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:19.884989 6935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8c4a55e09d41e391b8ba318dfe6896" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c"
I20240502 01:54:19.885061 6802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8c4a55e09d41e391b8ba318dfe6896" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
I20240502 01:54:19.885249 6802 raft_consensus.cc:3037] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.885224 6935 raft_consensus.cc:3037] T 7d8c4a55e09d41e391b8ba318dfe6896 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:19.887226 6802 raft_consensus.cc:2454] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 1.
I20240502 01:54:19.887478 6935 raft_consensus.cc:2454] T 7d8c4a55e09d41e391b8ba318dfe6896 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 1.
I20240502 01:54:19.887619 7133 leader_election.cc:302] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8214b6c4de6d4aa58a22af551eb8dedd, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:19.887909 8420 raft_consensus.cc:2781] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:19.888060 8420 raft_consensus.cc:686] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Becoming Leader. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Running, Role: LEADER
I20240502 01:54:19.888362 8420 consensus_queue.cc:237] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:19.891076 6652 catalog_manager.cc:5484] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc21155d36d5416fa3650f0268a20b17 (127.31.161.196). New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:20.106345 8246 consensus_queue.cc:1035] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:20.114004 8472 consensus_queue.cc:1035] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:20.187284 8399 consensus_queue.cc:1035] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:20.199909 8399 consensus_queue.cc:1035] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:20.326339 8479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:20.326728 8479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:20.336359 8479 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240502 01:54:20.384440 8240 consensus_queue.cc:1035] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:20.388967 8420 consensus_queue.cc:1035] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 225.783 ms
time per row: 0.112892 ms
I20240502 01:54:20.717244 6656 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43140:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:20.726683 7046 tablet_service.cc:1467] Processing CreateTablet for tablet ee804f1981f84a118051996fb83056b8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5b2a4358eba544959466c46759744194]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.726739 7179 tablet_service.cc:1467] Processing CreateTablet for tablet ee804f1981f84a118051996fb83056b8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5b2a4358eba544959466c46759744194]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.727173 7046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee804f1981f84a118051996fb83056b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.727176 7179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee804f1981f84a118051996fb83056b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.728399 7047 tablet_service.cc:1467] Processing CreateTablet for tablet 945224f5cff74edb9447bebc743e5086 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5b2a4358eba544959466c46759744194]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.728824 7047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 945224f5cff74edb9447bebc743e5086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.728966 6915 tablet_service.cc:1467] Processing CreateTablet for tablet 4c1d0974a2fd4a548806b6b6560a89f8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5b2a4358eba544959466c46759744194]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.729393 6915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c1d0974a2fd4a548806b6b6560a89f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.733906 6784 tablet_service.cc:1467] Processing CreateTablet for tablet ee804f1981f84a118051996fb83056b8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5b2a4358eba544959466c46759744194]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.734012 6783 tablet_service.cc:1467] Processing CreateTablet for tablet 945224f5cff74edb9447bebc743e5086 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5b2a4358eba544959466c46759744194]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.734364 6784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee804f1981f84a118051996fb83056b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.734599 6782 tablet_service.cc:1467] Processing CreateTablet for tablet 4c1d0974a2fd4a548806b6b6560a89f8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5b2a4358eba544959466c46759744194]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.734948 6782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c1d0974a2fd4a548806b6b6560a89f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.735416 7178 tablet_service.cc:1467] Processing CreateTablet for tablet 945224f5cff74edb9447bebc743e5086 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5b2a4358eba544959466c46759744194]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.735772 7178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 945224f5cff74edb9447bebc743e5086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.728369 7048 tablet_service.cc:1467] Processing CreateTablet for tablet 4c1d0974a2fd4a548806b6b6560a89f8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5b2a4358eba544959466c46759744194]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:20.737589 7048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c1d0974a2fd4a548806b6b6560a89f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.740926 6783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 945224f5cff74edb9447bebc743e5086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:20.743598 8544 tablet_bootstrap.cc:492] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:20.745543 8544 tablet_bootstrap.cc:654] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.747692 8545 tablet_bootstrap.cc:492] T ee804f1981f84a118051996fb83056b8 P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:20.749547 8545 tablet_bootstrap.cc:654] T ee804f1981f84a118051996fb83056b8 P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.751212 8542 tablet_bootstrap.cc:492] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17: Bootstrap starting.
I20240502 01:54:20.753266 8544 tablet_bootstrap.cc:492] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:20.753508 8544 ts_tablet_manager.cc:1367] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20240502 01:54:20.753670 8542 tablet_bootstrap.cc:654] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.754590 8544 raft_consensus.cc:348] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:20.755044 8544 raft_consensus.cc:374] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.755198 8544 raft_consensus.cc:729] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.755543 8544 consensus_queue.cc:260] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:20.756465 8544 ts_tablet_manager.cc:1398] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:20.756831 8544 tablet_bootstrap.cc:492] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:20.758925 8544 tablet_bootstrap.cc:654] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.772483 8544 tablet_bootstrap.cc:492] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:20.772753 8544 ts_tablet_manager.cc:1367] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.007s
I20240502 01:54:20.775177 8544 raft_consensus.cc:348] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:20.775521 8544 raft_consensus.cc:374] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.775640 8544 raft_consensus.cc:729] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.776270 8544 consensus_queue.cc:260] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:20.777053 8544 ts_tablet_manager.cc:1398] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:54:20.779165 8475 raft_consensus.cc:482] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:20.779302 8475 raft_consensus.cc:504] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:20.780053 8475 leader_election.cc:288] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873)
I20240502 01:54:20.782248 7199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee804f1981f84a118051996fb83056b8" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17" is_pre_election: true
W20240502 01:54:20.783181 6737 leader_election.cc:341] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:20.783593 7066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee804f1981f84a118051996fb83056b8" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3" is_pre_election: true
W20240502 01:54:20.784250 6738 leader_election.cc:341] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:54:20.784411 6738 leader_election.cc:302] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8214b6c4de6d4aa58a22af551eb8dedd; no voters: 11c097d7d32a4c22a6385a7465876eb3, fc21155d36d5416fa3650f0268a20b17
I20240502 01:54:20.784904 8475 raft_consensus.cc:2726] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:54:20.787003 8544 tablet_bootstrap.cc:492] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd: Bootstrap starting.
I20240502 01:54:20.788882 8544 tablet_bootstrap.cc:654] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.797047 8543 tablet_bootstrap.cc:492] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c: Bootstrap starting.
I20240502 01:54:20.797906 8542 tablet_bootstrap.cc:492] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17: No bootstrap required, opened a new log
I20240502 01:54:20.798069 8544 tablet_bootstrap.cc:492] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd: No bootstrap required, opened a new log
I20240502 01:54:20.798079 8542 ts_tablet_manager.cc:1367] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17: Time spent bootstrapping tablet: real 0.047s user 0.009s sys 0.014s
I20240502 01:54:20.798213 8544 ts_tablet_manager.cc:1367] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240502 01:54:20.798884 8544 raft_consensus.cc:348] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:20.798866 8542 raft_consensus.cc:348] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:20.799084 8542 raft_consensus.cc:374] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.799084 8544 raft_consensus.cc:374] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.799180 8542 raft_consensus.cc:729] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.799232 8544 raft_consensus.cc:729] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.799497 8544 consensus_queue.cc:260] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:20.800194 8544 ts_tablet_manager.cc:1398] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:54:20.800617 8545 tablet_bootstrap.cc:492] T ee804f1981f84a118051996fb83056b8 P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:20.800778 8545 ts_tablet_manager.cc:1367] T ee804f1981f84a118051996fb83056b8 P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.022s
I20240502 01:54:20.801277 8542 consensus_queue.cc:260] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:20.801479 8545 raft_consensus.cc:348] T ee804f1981f84a118051996fb83056b8 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:20.801662 8545 raft_consensus.cc:374] T ee804f1981f84a118051996fb83056b8 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.801746 8545 raft_consensus.cc:729] T ee804f1981f84a118051996fb83056b8 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.802002 8545 consensus_queue.cc:260] T ee804f1981f84a118051996fb83056b8 P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:20.802681 8545 ts_tablet_manager.cc:1398] T ee804f1981f84a118051996fb83056b8 P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:20.803122 8545 tablet_bootstrap.cc:492] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:20.803189 8542 ts_tablet_manager.cc:1398] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240502 01:54:20.803659 8542 tablet_bootstrap.cc:492] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17: Bootstrap starting.
I20240502 01:54:20.804644 8545 tablet_bootstrap.cc:654] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.805240 8542 tablet_bootstrap.cc:654] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.810473 8543 tablet_bootstrap.cc:654] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.812986 8543 tablet_bootstrap.cc:492] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c: No bootstrap required, opened a new log
I20240502 01:54:20.813177 8542 tablet_bootstrap.cc:492] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17: No bootstrap required, opened a new log
I20240502 01:54:20.813285 8543 ts_tablet_manager.cc:1367] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.004s
I20240502 01:54:20.813334 8542 ts_tablet_manager.cc:1367] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20240502 01:54:20.814097 8542 raft_consensus.cc:348] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:20.814105 8543 raft_consensus.cc:348] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:20.814327 8542 raft_consensus.cc:374] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.814405 8542 raft_consensus.cc:729] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.814437 8543 raft_consensus.cc:374] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.814525 8543 raft_consensus.cc:729] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.814662 8542 consensus_queue.cc:260] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:20.814800 8543 consensus_queue.cc:260] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:20.815336 8543 ts_tablet_manager.cc:1398] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:54:20.815371 8542 ts_tablet_manager.cc:1398] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:20.816920 8545 tablet_bootstrap.cc:492] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:20.817111 8545 ts_tablet_manager.cc:1367] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.002s
I20240502 01:54:20.819058 8545 raft_consensus.cc:348] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:20.819283 8545 raft_consensus.cc:374] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.819343 8545 raft_consensus.cc:729] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.819522 8545 consensus_queue.cc:260] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:20.819919 8545 ts_tablet_manager.cc:1398] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 01:54:20.820317 8545 tablet_bootstrap.cc:492] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3: Bootstrap starting.
I20240502 01:54:20.821574 8545 tablet_bootstrap.cc:654] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:20.823283 8545 tablet_bootstrap.cc:492] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3: No bootstrap required, opened a new log
I20240502 01:54:20.823431 8545 ts_tablet_manager.cc:1367] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:20.824151 8545 raft_consensus.cc:348] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:20.824308 8545 raft_consensus.cc:374] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:20.824395 8545 raft_consensus.cc:729] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Initialized, Role: FOLLOWER
I20240502 01:54:20.824595 8545 consensus_queue.cc:260] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:20.825136 8545 ts_tablet_manager.cc:1398] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:20.837648 8420 raft_consensus.cc:482] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:20.837819 8420 raft_consensus.cc:504] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:20.838357 8420 leader_election.cc:288] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:20.838814 7066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee804f1981f84a118051996fb83056b8" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3" is_pre_election: true
I20240502 01:54:20.838853 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee804f1981f84a118051996fb83056b8" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" is_pre_election: true
I20240502 01:54:20.839037 7066 raft_consensus.cc:2454] T ee804f1981f84a118051996fb83056b8 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 0.
I20240502 01:54:20.839051 6804 raft_consensus.cc:2454] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 0.
I20240502 01:54:20.839476 7133 leader_election.cc:302] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 8214b6c4de6d4aa58a22af551eb8dedd, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:20.839769 8420 raft_consensus.cc:2781] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:20.839875 8420 raft_consensus.cc:482] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:20.839984 8420 raft_consensus.cc:3037] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.841825 8420 raft_consensus.cc:504] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:20.842404 8420 leader_election.cc:288] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 election: Requested vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:20.842797 7066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee804f1981f84a118051996fb83056b8" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
I20240502 01:54:20.842864 6804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee804f1981f84a118051996fb83056b8" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
I20240502 01:54:20.843012 7066 raft_consensus.cc:3037] T ee804f1981f84a118051996fb83056b8 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.843025 6804 raft_consensus.cc:3037] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.844798 7066 raft_consensus.cc:2454] T ee804f1981f84a118051996fb83056b8 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 1.
I20240502 01:54:20.844810 6804 raft_consensus.cc:2454] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 1.
I20240502 01:54:20.845186 7133 leader_election.cc:302] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8214b6c4de6d4aa58a22af551eb8dedd, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:20.845463 8420 raft_consensus.cc:2781] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:20.845613 8420 raft_consensus.cc:686] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Becoming Leader. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Running, Role: LEADER
I20240502 01:54:20.845839 8420 consensus_queue.cc:237] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } }
I20240502 01:54:20.848814 6654 catalog_manager.cc:5484] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc21155d36d5416fa3650f0268a20b17 (127.31.161.196). New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:20.869303 8475 raft_consensus.cc:482] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:20.869484 8475 raft_consensus.cc:504] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:20.870218 8475 leader_election.cc:288] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873)
I20240502 01:54:20.870644 7066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945224f5cff74edb9447bebc743e5086" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3" is_pre_election: true
I20240502 01:54:20.870756 7199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945224f5cff74edb9447bebc743e5086" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17" is_pre_election: true
I20240502 01:54:20.870883 7066 raft_consensus.cc:2454] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 0.
I20240502 01:54:20.871026 7199 raft_consensus.cc:2454] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 0.
I20240502 01:54:20.871315 6738 leader_election.cc:302] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 8214b6c4de6d4aa58a22af551eb8dedd; no voters:
I20240502 01:54:20.871608 8475 raft_consensus.cc:2781] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:20.871716 8475 raft_consensus.cc:482] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:20.871795 8475 raft_consensus.cc:3037] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.873790 8475 raft_consensus.cc:504] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:20.874307 8475 leader_election.cc:288] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 election: Requested vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873)
I20240502 01:54:20.874737 7199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945224f5cff74edb9447bebc743e5086" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc21155d36d5416fa3650f0268a20b17"
I20240502 01:54:20.874799 7066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "945224f5cff74edb9447bebc743e5086" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
I20240502 01:54:20.874958 7199 raft_consensus.cc:3037] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.875016 7066 raft_consensus.cc:3037] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.877825 7199 raft_consensus.cc:2454] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 1.
I20240502 01:54:20.877848 7066 raft_consensus.cc:2454] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 1.
I20240502 01:54:20.878285 6737 leader_election.cc:302] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8214b6c4de6d4aa58a22af551eb8dedd, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:20.878618 8475 raft_consensus.cc:2781] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:20.878784 8475 raft_consensus.cc:686] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Becoming Leader. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Running, Role: LEADER
I20240502 01:54:20.879101 8475 consensus_queue.cc:237] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } }
I20240502 01:54:20.881930 6654 catalog_manager.cc:5484] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193). New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:20.975783 8475 raft_consensus.cc:482] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:54:20.976008 8475 raft_consensus.cc:504] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:20.976739 8475 leader_election.cc:288] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:20.977465 7066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c1d0974a2fd4a548806b6b6560a89f8" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3" is_pre_election: true
I20240502 01:54:20.977483 6935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c1d0974a2fd4a548806b6b6560a89f8" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c" is_pre_election: true
I20240502 01:54:20.977727 7066 raft_consensus.cc:2454] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 0.
I20240502 01:54:20.977762 6935 raft_consensus.cc:2454] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 0.
I20240502 01:54:20.978142 6738 leader_election.cc:302] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 8214b6c4de6d4aa58a22af551eb8dedd; no voters:
I20240502 01:54:20.978437 8475 raft_consensus.cc:2781] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:54:20.978538 8475 raft_consensus.cc:482] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:54:20.978680 8475 raft_consensus.cc:3037] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.980562 8475 raft_consensus.cc:504] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:20.981140 8475 leader_election.cc:288] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 election: Requested vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:20.981575 7066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c1d0974a2fd4a548806b6b6560a89f8" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
I20240502 01:54:20.981616 6935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c1d0974a2fd4a548806b6b6560a89f8" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137e2183ff57411791e9af58dfab0a3c"
I20240502 01:54:20.981799 7066 raft_consensus.cc:3037] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.981802 6935 raft_consensus.cc:3037] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:20.984494 7066 raft_consensus.cc:2454] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 1.
I20240502 01:54:20.984494 6935 raft_consensus.cc:2454] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 1.
I20240502 01:54:20.984872 6738 leader_election.cc:302] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 137e2183ff57411791e9af58dfab0a3c, 8214b6c4de6d4aa58a22af551eb8dedd; no voters:
I20240502 01:54:20.985126 8475 raft_consensus.cc:2781] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:20.985297 8475 raft_consensus.cc:686] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Becoming Leader. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Running, Role: LEADER
I20240502 01:54:20.985589 8475 consensus_queue.cc:237] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } }
I20240502 01:54:20.988633 6654 catalog_manager.cc:5484] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193). New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:21.287452 8420 consensus_queue.cc:1035] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:21.297420 8240 consensus_queue.cc:1035] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:21.317286 8475 consensus_queue.cc:1035] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:21.324158 8472 consensus_queue.cc:1035] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:54:21.361009 8558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:21.361380 8558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:21.369187 8558 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 01:54:21.420374 8475 consensus_queue.cc:1035] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:54:21.430843 8472 consensus_queue.cc:1035] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 250.004 ms
time per row: 0.125002 ms
I20240502 01:54:21.771911 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.197:44451
--local_ip_for_outbound_sockets=127.31.161.197
--tserver_master_addrs=127.31.161.254:36997
--webserver_port=41839
--webserver_interface=127.31.161.197
--builtin_ntp_servers=127.31.161.212:43969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 01:54:22.114085 8609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:22.114358 8609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:22.114468 8609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:22.121941 8609 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:22.122051 8609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:22.122169 8609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.197
I20240502 01:54:22.131595 8609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:43969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.197:44451
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.31.161.197
--webserver_port=41839
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.161.254:36997
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.197
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:22.133021 8609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:22.135113 8609 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:22.148130 8614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:22.148159 8617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:22.148627 8615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:22.149060 8609 server_base.cc:1034] running on GCE node
I20240502 01:54:22.149693 8609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:22.150631 8609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:22.151830 8609 hybrid_clock.cc:648] HybridClock initialized: now 1714614862151781 us; error 44 us; skew 500 ppm
I20240502 01:54:22.152144 8609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:22.154253 8609 webserver.cc:460] Webserver started at http://127.31.161.197:41839/ using document root <none> and password file <none>
I20240502 01:54:22.155037 8609 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:22.155217 8609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:22.160781 8609 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 01:54:22.163558 8623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:22.164824 8609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:22.164988 8609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "3f156b7cfb4f40f481699b257153e5cb"
format_stamp: "Formatted at 2024-05-02 01:54:04 on dist-test-slave-06l8"
I20240502 01:54:22.165814 8609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:22.199028 8609 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:22.200001 8609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:22.200311 8609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:22.201406 8609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:22.203419 8609 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:22.203544 8609 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:22.203673 8609 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:22.203737 8609 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:22.233668 8609 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.197:44451
I20240502 01:54:22.233729 8736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.197:44451 every 8 connection(s)
I20240502 01:54:22.235093 8609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:54:22.243355 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 8609
I20240502 01:54:22.243947 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.198:37123
--local_ip_for_outbound_sockets=127.31.161.198
--tserver_master_addrs=127.31.161.254:36997
--webserver_port=46173
--webserver_interface=127.31.161.198
--builtin_ntp_servers=127.31.161.212:43969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:54:22.247227 8737 heartbeater.cc:347] Connected to a master server at 127.31.161.254:36997
I20240502 01:54:22.247550 8737 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:22.248523 8737 heartbeater.cc:511] Master 127.31.161.254:36997 requested a full tablet report, sending...
I20240502 01:54:22.250288 6654 ts_manager.cc:194] Re-registered known tserver with Master: 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197:44451)
I20240502 01:54:22.251359 6654 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.197:59601
W20240502 01:54:22.595342 8741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:22.595731 8741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:22.595875 8741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:22.603331 8741 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:22.603463 8741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:22.603618 8741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.198
I20240502 01:54:22.613641 8741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:43969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.198:37123
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.31.161.198
--webserver_port=46173
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.161.254:36997
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.198
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:22.614924 8741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:22.616637 8741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:22.627866 8747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:22.628410 8750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:22.628820 8748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:22.629209 8741 server_base.cc:1034] running on GCE node
I20240502 01:54:22.629750 8741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:22.630697 8741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:22.631894 8741 hybrid_clock.cc:648] HybridClock initialized: now 1714614862631846 us; error 35 us; skew 500 ppm
I20240502 01:54:22.632193 8741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:22.634393 8741 webserver.cc:460] Webserver started at http://127.31.161.198:46173/ using document root <none> and password file <none>
I20240502 01:54:22.635082 8741 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:22.635205 8741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:22.640508 8741 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 01:54:22.643273 8756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:22.644593 8741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:22.644738 8741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9e47e5c098b848c9abe1559cbcbb6321"
format_stamp: "Formatted at 2024-05-02 01:54:04 on dist-test-slave-06l8"
I20240502 01:54:22.645808 8741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:22.660979 8741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:22.661927 8741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:22.662212 8741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:22.663123 8741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:22.664690 8741 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:22.664878 8741 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:22.665107 8741 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:22.665201 8741 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240502 01:54:22.695969 8741 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.198:37123
I20240502 01:54:22.696036 8869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.198:37123 every 8 connection(s)
I20240502 01:54:22.697499 8741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:54:22.707034 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 8741
I20240502 01:54:22.707631 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.199:35245
--local_ip_for_outbound_sockets=127.31.161.199
--tserver_master_addrs=127.31.161.254:36997
--webserver_port=38573
--webserver_interface=127.31.161.199
--builtin_ntp_servers=127.31.161.212:43969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:54:22.710750 8870 heartbeater.cc:347] Connected to a master server at 127.31.161.254:36997
I20240502 01:54:22.711035 8870 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:22.711817 8870 heartbeater.cc:511] Master 127.31.161.254:36997 requested a full tablet report, sending...
I20240502 01:54:22.713196 6654 ts_manager.cc:194] Re-registered known tserver with Master: 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198:37123)
I20240502 01:54:22.714166 6654 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.198:46169
W20240502 01:54:23.052553 8873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:23.052819 8873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:23.052958 8873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:23.060281 8873 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:23.060401 8873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:23.060554 8873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.199
I20240502 01:54:23.070220 8873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:43969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.199:35245
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.31.161.199
--webserver_port=38573
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.161.254:36997
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.199
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:23.071784 8873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:23.073657 8873 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:23.086020 8879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:23.086683 8880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:23.087299 8882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:23.087518 8873 server_base.cc:1034] running on GCE node
I20240502 01:54:23.088092 8873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:23.088958 8873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:23.090162 8873 hybrid_clock.cc:648] HybridClock initialized: now 1714614863090117 us; error 58 us; skew 500 ppm
I20240502 01:54:23.090471 8873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:23.092660 8873 webserver.cc:460] Webserver started at http://127.31.161.199:38573/ using document root <none> and password file <none>
I20240502 01:54:23.093515 8873 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:23.093668 8873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:23.099691 8873 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20240502 01:54:23.102664 8889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:23.103937 8873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:54:23.104082 8873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "c39b0639d5fb4d9da09bf1d67d617fcc"
format_stamp: "Formatted at 2024-05-02 01:54:04 on dist-test-slave-06l8"
I20240502 01:54:23.104905 8873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:23.122488 8873 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:23.123258 8873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:23.123589 8873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:23.124473 8873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:23.125932 8873 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:23.126003 8873 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:23.126093 8873 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:23.126134 8873 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:23.154361 8873 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.199:35245
I20240502 01:54:23.154407 9001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.199:35245 every 8 connection(s)
I20240502 01:54:23.155771 8873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:54:23.159682 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 8873
I20240502 01:54:23.160441 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.200:34863
--local_ip_for_outbound_sockets=127.31.161.200
--tserver_master_addrs=127.31.161.254:36997
--webserver_port=38861
--webserver_interface=127.31.161.200
--builtin_ntp_servers=127.31.161.212:43969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:54:23.171149 9002 heartbeater.cc:347] Connected to a master server at 127.31.161.254:36997
I20240502 01:54:23.171506 9002 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:23.172536 9002 heartbeater.cc:511] Master 127.31.161.254:36997 requested a full tablet report, sending...
I20240502 01:54:23.174330 6654 ts_manager.cc:194] Re-registered known tserver with Master: c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199:35245)
I20240502 01:54:23.175192 6654 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.199:60365
I20240502 01:54:23.254415 8737 heartbeater.cc:503] Master 127.31.161.254:36997 was elected leader, sending a full tablet report...
W20240502 01:54:23.513278 9005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:23.513551 9005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:23.513656 9005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:23.521215 9005 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:23.521361 9005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:23.521526 9005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.200
I20240502 01:54:23.531239 9005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:43969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.200:34863
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.31.161.200
--webserver_port=38861
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.161.254:36997
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.200
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:23.532539 9005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:23.534319 9005 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:23.545544 9010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:23.545518 9011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:23.546208 9013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:23.546591 9005 server_base.cc:1034] running on GCE node
I20240502 01:54:23.547142 9005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:23.548015 9005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:23.549219 9005 hybrid_clock.cc:648] HybridClock initialized: now 1714614863549172 us; error 41 us; skew 500 ppm
I20240502 01:54:23.549518 9005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:23.551936 9005 webserver.cc:460] Webserver started at http://127.31.161.200:38861/ using document root <none> and password file <none>
I20240502 01:54:23.552753 9005 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:23.552917 9005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:23.558703 9005 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 01:54:23.561414 9019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:23.562698 9005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240502 01:54:23.562830 9005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "48866680637c40368e45ba844efb9298"
format_stamp: "Formatted at 2024-05-02 01:54:05 on dist-test-slave-06l8"
I20240502 01:54:23.563594 9005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:23.594699 9005 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:23.595533 9005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:23.595804 9005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:23.596768 9005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:23.598394 9005 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:23.598471 9005 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:23.598594 9005 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:23.598645 9005 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:23.628857 9005 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.200:34863
I20240502 01:54:23.628902 9131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.200:34863 every 8 connection(s)
I20240502 01:54:23.630405 9005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:54:23.635659 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 9005
I20240502 01:54:23.644843 9132 heartbeater.cc:347] Connected to a master server at 127.31.161.254:36997
I20240502 01:54:23.645184 9132 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:23.646139 9132 heartbeater.cc:511] Master 127.31.161.254:36997 requested a full tablet report, sending...
I20240502 01:54:23.647666 6654 ts_manager.cc:194] Re-registered known tserver with Master: 48866680637c40368e45ba844efb9298 (127.31.161.200:34863)
I20240502 01:54:23.648713 6654 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.200:44991
I20240502 01:54:23.717057 8870 heartbeater.cc:503] Master 127.31.161.254:36997 was elected leader, sending a full tablet report...
W20240502 01:54:24.118760 9136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:24.119155 9136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:24.131304 9136 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:54:24.178287 9002 heartbeater.cc:503] Master 127.31.161.254:36997 was elected leader, sending a full tablet report...
I20240502 01:54:24.385120 7047 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:24.399097 9067 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:24.426808 7047 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:24.446555 6915 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:24.454011 6782 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:24.462270 6915 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:24.480901 8672 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:24.508920 9067 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:24.535928 6782 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:24.570060 8672 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:24.612421 8937 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:24.624539 8804 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:24.639308 8804 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:24.651616 9132 heartbeater.cc:503] Master 127.31.161.254:36997 was elected leader, sending a full tablet report...
I20240502 01:54:24.658977 8937 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:24.670630 7179 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:54:24.692909 7179 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:54:25.013356 7047 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:25.025656 8672 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:25.031476 6915 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:25.052059 9067 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:25.090602 8937 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:25.091763 8804 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:25.119814 7179 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:54:25.133317 6782 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:25.409605 6915 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:25.414660 7047 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:25.420341 9067 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:25.446659 8937 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:25.447109 6782 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:25.450383 8804 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:25.461431 7179 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:54:25.462061 8672 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 01:54:25.566395 9272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:25.566838 9272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:25.578899 9272 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:54:25.611021 7199 consensus_queue.cc:237] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:25.615445 6804 raft_consensus.cc:1261] T dabfafc485c144b995aa4248fb1e897b P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:25.617987 9274 consensus_queue.cc:1035] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240502 01:54:25.619040 7067 raft_consensus.cc:1261] T dabfafc485c144b995aa4248fb1e897b P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:25.620649 9274 consensus_queue.cc:1035] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:25.626482 9138 raft_consensus.cc:2932] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:25.632364 6804 raft_consensus.cc:2932] T dabfafc485c144b995aa4248fb1e897b P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
W20240502 01:54:25.642910 7134 consensus_peers.cc:485] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 -> Peer c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199:35245): Couldn't send request to peer c39b0639d5fb4d9da09bf1d67d617fcc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dabfafc485c144b995aa4248fb1e897b. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:25.646672 6656 catalog_manager.cc:5484] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:25.663064 7067 raft_consensus.cc:2932] T dabfafc485c144b995aa4248fb1e897b P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:25.675595 7199 consensus_queue.cc:237] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:25.678531 7067 raft_consensus.cc:1261] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:25.678592 6936 raft_consensus.cc:1261] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:25.679930 9138 consensus_queue.cc:1035] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 01:54:25.681537 9348 consensus_queue.cc:1035] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:25.687918 7134 consensus_peers.cc:485] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 -> Peer 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198:37123): Couldn't send request to peer 9e47e5c098b848c9abe1559cbcbb6321. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5acc7c2405464a328b2016bb2293f7e8. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:25.690980 9348 raft_consensus.cc:2932] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:25.691529 7067 raft_consensus.cc:2932] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:25.706456 6659 catalog_manager.cc:5484] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:25.710918 6936 raft_consensus.cc:2932] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:25.731803 7199 consensus_queue.cc:237] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:25.734573 6936 raft_consensus.cc:1261] T 7d8c4a55e09d41e391b8ba318dfe6896 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:25.734719 6804 raft_consensus.cc:1261] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:25.735324 9274 consensus_queue.cc:1035] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:25.735901 9138 consensus_queue.cc:1035] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:25.738476 9138 raft_consensus.cc:2932] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:25.742480 6804 raft_consensus.cc:2932] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
W20240502 01:54:25.743793 7135 consensus_peers.cc:485] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 -> Peer 48866680637c40368e45ba844efb9298 (127.31.161.200:34863): Couldn't send request to peer 48866680637c40368e45ba844efb9298. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d8c4a55e09d41e391b8ba318dfe6896. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:25.745522 6659 catalog_manager.cc:5484] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:25.758522 6936 raft_consensus.cc:2932] T 7d8c4a55e09d41e391b8ba318dfe6896 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:25.794948 6936 consensus_queue.cc:237] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:25.802264 7199 raft_consensus.cc:1261] T db7a48f131f141f0a4e45036434f8129 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:25.802904 6804 raft_consensus.cc:1261] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:25.803704 9369 consensus_queue.cc:1035] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:25.804025 8887 consensus_queue.cc:1035] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:25.837466 6871 consensus_peers.cc:485] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c -> Peer 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197:44451): Couldn't send request to peer 3f156b7cfb4f40f481699b257153e5cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db7a48f131f141f0a4e45036434f8129. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:25.843966 9369 raft_consensus.cc:2932] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:25.851166 6804 raft_consensus.cc:2932] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:25.851612 7199 raft_consensus.cc:2932] T db7a48f131f141f0a4e45036434f8129 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:25.855321 6654 catalog_manager.cc:5484] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c reported cstate change: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:25.870425 7067 consensus_queue.cc:237] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:25.873947 6804 raft_consensus.cc:1261] T c3397a96d989416fa2bccdc71c7ea9ec P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:25.874562 9407 consensus_queue.cc:1035] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:25.876914 6936 raft_consensus.cc:1261] T c3397a96d989416fa2bccdc71c7ea9ec P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:25.877548 9407 consensus_queue.cc:1035] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:25.887404 7003 consensus_peers.cc:485] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 -> Peer c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199:35245): Couldn't send request to peer c39b0639d5fb4d9da09bf1d67d617fcc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3397a96d989416fa2bccdc71c7ea9ec. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:25.888365 8399 raft_consensus.cc:2932] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:25.891568 6936 raft_consensus.cc:2932] T c3397a96d989416fa2bccdc71c7ea9ec P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:25.894937 6802 raft_consensus.cc:2932] T c3397a96d989416fa2bccdc71c7ea9ec P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:25.895316 6654 catalog_manager.cc:5484] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:25.927642 8672 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:25.929563 9067 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:25.930831 6915 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:25.962878 6936 consensus_queue.cc:237] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:25.973322 7047 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:25.995018 7199 raft_consensus.cc:1261] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:25.996075 8887 consensus_queue.cc:1035] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.001183 7067 raft_consensus.cc:1261] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.002053 8887 consensus_queue.cc:1035] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:26.006428 6871 consensus_peers.cc:485] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c -> Peer 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198:37123): Couldn't send request to peer 9e47e5c098b848c9abe1559cbcbb6321. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01e98643312b4a2eb80e3bcb3d95674c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:26.010389 6782 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:26.014428 8937 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:26.033613 8804 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:26.050828 9369 raft_consensus.cc:2932] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:26.052796 7179 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:54:26.053407 7199 raft_consensus.cc:2932] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:26.057514 6654 catalog_manager.cc:5484] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c reported cstate change: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:26.091315 7066 raft_consensus.cc:2932] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:26.110133 7067 consensus_queue.cc:237] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:26.113299 6936 raft_consensus.cc:1261] T f6a165cbc26f4defb70866779f83bfdc P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.114004 9407 consensus_queue.cc:1035] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.121238 6802 raft_consensus.cc:1261] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.122359 8399 consensus_queue.cc:1035] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:26.128007 7004 consensus_peers.cc:485] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 -> Peer 48866680637c40368e45ba844efb9298 (127.31.161.200:34863): Couldn't send request to peer 48866680637c40368e45ba844efb9298. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6a165cbc26f4defb70866779f83bfdc. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:26.144129 6802 raft_consensus.cc:2932] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:26.143596 9407 raft_consensus.cc:2932] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:26.151121 6654 catalog_manager.cc:5484] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:26.160773 6936 raft_consensus.cc:2932] T f6a165cbc26f4defb70866779f83bfdc P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
Runtime error: ksck discovered errors
I20240502 01:54:26.186638 6802 consensus_queue.cc:237] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:26.201692 9443 ts_tablet_manager.cc:902] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298: Initiating tablet copy from peer fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873)
I20240502 01:54:26.207332 7198 raft_consensus.cc:1261] T f03b37fd2f844f61a39daadc698c9b29 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.207993 9288 consensus_queue.cc:1035] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.212774 9443 tablet_copy_client.cc:308] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:38873
I20240502 01:54:26.226794 9288 raft_consensus.cc:2932] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:26.228974 9453 ts_tablet_manager.cc:902] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321: Initiating tablet copy from peer fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873)
I20240502 01:54:26.235239 7198 raft_consensus.cc:2932] T f03b37fd2f844f61a39daadc698c9b29 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:26.236061 6659 catalog_manager.cc:5484] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:26.240187 6935 raft_consensus.cc:1261] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.249598 8246 consensus_queue.cc:1035] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.252084 6935 raft_consensus.cc:2932] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:26.255898 7219 tablet_copy_service.cc:140] P fc21155d36d5416fa3650f0268a20b17: Received BeginTabletCopySession request for tablet 7d8c4a55e09d41e391b8ba318dfe6896 from peer 48866680637c40368e45ba844efb9298 ({username='slave'} at 127.31.161.200:55615)
I20240502 01:54:26.256157 7219 tablet_copy_service.cc:161] P fc21155d36d5416fa3650f0268a20b17: Beginning new tablet copy session on tablet 7d8c4a55e09d41e391b8ba318dfe6896 from peer 48866680637c40368e45ba844efb9298 at {username='slave'} at 127.31.161.200:55615: session id = 48866680637c40368e45ba844efb9298-7d8c4a55e09d41e391b8ba318dfe6896
I20240502 01:54:26.259984 7219 tablet_copy_source_session.cc:205] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 01:54:26.266566 6738 consensus_peers.cc:485] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd -> Peer c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199:35245): Couldn't send request to peer c39b0639d5fb4d9da09bf1d67d617fcc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f03b37fd2f844f61a39daadc698c9b29. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:26.267145 9443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d8c4a55e09d41e391b8ba318dfe6896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.261920 9446 ts_tablet_manager.cc:902] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb: Initiating tablet copy from peer 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:26.289884 9455 ts_tablet_manager.cc:902] T dabfafc485c144b995aa4248fb1e897b P c39b0639d5fb4d9da09bf1d67d617fcc: Initiating tablet copy from peer fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873)
I20240502 01:54:26.290526 9443 tablet_copy_client.cc:783] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:26.291216 9443 tablet_copy_client.cc:647] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:26.294407 9450 tablet_copy_client.cc:1002] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:26.313271 9446 tablet_copy_client.cc:308] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.194:45315
I20240502 01:54:26.325711 9453 tablet_copy_client.cc:308] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:38873
I20240502 01:54:26.326444 9455 tablet_copy_client.cc:308] T dabfafc485c144b995aa4248fb1e897b P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:38873
I20240502 01:54:26.337611 9476 ts_tablet_manager.cc:902] T c3397a96d989416fa2bccdc71c7ea9ec P c39b0639d5fb4d9da09bf1d67d617fcc: Initiating tablet copy from peer 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:26.346836 6958 tablet_copy_service.cc:140] P 137e2183ff57411791e9af58dfab0a3c: Received BeginTabletCopySession request for tablet db7a48f131f141f0a4e45036434f8129 from peer 3f156b7cfb4f40f481699b257153e5cb ({username='slave'} at 127.31.161.197:41393)
I20240502 01:54:26.347093 6958 tablet_copy_service.cc:161] P 137e2183ff57411791e9af58dfab0a3c: Beginning new tablet copy session on tablet db7a48f131f141f0a4e45036434f8129 from peer 3f156b7cfb4f40f481699b257153e5cb at {username='slave'} at 127.31.161.197:41393: session id = 3f156b7cfb4f40f481699b257153e5cb-db7a48f131f141f0a4e45036434f8129
I20240502 01:54:26.351027 6958 tablet_copy_source_session.cc:205] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:26.356320 7219 tablet_copy_service.cc:140] P fc21155d36d5416fa3650f0268a20b17: Received BeginTabletCopySession request for tablet 5acc7c2405464a328b2016bb2293f7e8 from peer 9e47e5c098b848c9abe1559cbcbb6321 ({username='slave'} at 127.31.161.198:52833)
I20240502 01:54:26.356551 7219 tablet_copy_service.cc:161] P fc21155d36d5416fa3650f0268a20b17: Beginning new tablet copy session on tablet 5acc7c2405464a328b2016bb2293f7e8 from peer 9e47e5c098b848c9abe1559cbcbb6321 at {username='slave'} at 127.31.161.198:52833: session id = 9e47e5c098b848c9abe1559cbcbb6321-5acc7c2405464a328b2016bb2293f7e8
I20240502 01:54:26.357730 7198 consensus_queue.cc:237] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:26.358062 9446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db7a48f131f141f0a4e45036434f8129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.360808 7219 tablet_copy_source_session.cc:205] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:26.365213 9476 tablet_copy_client.cc:308] T c3397a96d989416fa2bccdc71c7ea9ec P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:46083
I20240502 01:54:26.365767 9453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5acc7c2405464a328b2016bb2293f7e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.373333 6802 raft_consensus.cc:1261] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.374956 9274 consensus_queue.cc:1035] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.376889 7219 tablet_copy_service.cc:140] P fc21155d36d5416fa3650f0268a20b17: Received BeginTabletCopySession request for tablet dabfafc485c144b995aa4248fb1e897b from peer c39b0639d5fb4d9da09bf1d67d617fcc ({username='slave'} at 127.31.161.199:57741)
I20240502 01:54:26.377041 7219 tablet_copy_service.cc:161] P fc21155d36d5416fa3650f0268a20b17: Beginning new tablet copy session on tablet dabfafc485c144b995aa4248fb1e897b from peer c39b0639d5fb4d9da09bf1d67d617fcc at {username='slave'} at 127.31.161.199:57741: session id = c39b0639d5fb4d9da09bf1d67d617fcc-dabfafc485c144b995aa4248fb1e897b
I20240502 01:54:26.379822 7219 tablet_copy_source_session.cc:205] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:26.382262 9446 tablet_copy_client.cc:783] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:26.382649 9446 tablet_copy_client.cc:647] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:26.385243 9466 tablet_copy_client.cc:1002] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:26.385046 9455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dabfafc485c144b995aa4248fb1e897b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.390547 9138 raft_consensus.cc:2932] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:26.395694 6803 raft_consensus.cc:2932] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:26.397146 9453 tablet_copy_client.cc:783] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:26.397477 9453 tablet_copy_client.cc:647] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:26.401731 6659 catalog_manager.cc:5484] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:26.404158 7089 tablet_copy_service.cc:140] P 11c097d7d32a4c22a6385a7465876eb3: Received BeginTabletCopySession request for tablet c3397a96d989416fa2bccdc71c7ea9ec from peer c39b0639d5fb4d9da09bf1d67d617fcc ({username='slave'} at 127.31.161.199:34537)
I20240502 01:54:26.404444 7089 tablet_copy_service.cc:161] P 11c097d7d32a4c22a6385a7465876eb3: Beginning new tablet copy session on tablet c3397a96d989416fa2bccdc71c7ea9ec from peer c39b0639d5fb4d9da09bf1d67d617fcc at {username='slave'} at 127.31.161.199:34537: session id = c39b0639d5fb4d9da09bf1d67d617fcc-c3397a96d989416fa2bccdc71c7ea9ec
I20240502 01:54:26.405828 7067 raft_consensus.cc:1261] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.408622 7089 tablet_copy_source_session.cc:205] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:26.410715 9476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3397a96d989416fa2bccdc71c7ea9ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.411834 9455 tablet_copy_client.cc:783] T dabfafc485c144b995aa4248fb1e897b P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:26.412159 9455 tablet_copy_client.cc:647] T dabfafc485c144b995aa4248fb1e897b P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:26.414266 9473 tablet_copy_client.cc:1002] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:26.414219 9138 consensus_queue.cc:1035] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.415139 9471 tablet_copy_client.cc:1002] T dabfafc485c144b995aa4248fb1e897b P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:26.416193 7067 raft_consensus.cc:2932] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
W20240502 01:54:26.417397 7134 consensus_peers.cc:485] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 -> Peer 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197:44451): Couldn't send request to peer 3f156b7cfb4f40f481699b257153e5cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d55da66aad2140d788704b056e476e99. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:26.419878 9476 tablet_copy_client.cc:783] T c3397a96d989416fa2bccdc71c7ea9ec P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:26.420132 9476 tablet_copy_client.cc:647] T c3397a96d989416fa2bccdc71c7ea9ec P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:26.422326 9483 tablet_copy_client.cc:1002] T c3397a96d989416fa2bccdc71c7ea9ec P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:26.425794 7198 consensus_queue.cc:237] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:26.428550 6935 raft_consensus.cc:1261] T ae02d485bc56444e994a56f759fbed1f P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:26.428596 7135 consensus_peers.cc:485] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 -> Peer 48866680637c40368e45ba844efb9298 (127.31.161.200:34863): Couldn't send request to peer 48866680637c40368e45ba844efb9298. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae02d485bc56444e994a56f759fbed1f. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:26.428192 7067 raft_consensus.cc:1261] T ae02d485bc56444e994a56f759fbed1f P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.429090 9484 consensus_queue.cc:1035] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.429395 9138 consensus_queue.cc:1035] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.438584 9274 raft_consensus.cc:2932] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:26.439126 6935 raft_consensus.cc:2932] T ae02d485bc56444e994a56f759fbed1f P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:26.439296 7067 raft_consensus.cc:2932] T ae02d485bc56444e994a56f759fbed1f P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:26.444957 6654 catalog_manager.cc:5484] T ae02d485bc56444e994a56f759fbed1f P 137e2183ff57411791e9af58dfab0a3c reported cstate change: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:26.472780 7067 consensus_queue.cc:237] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:26.475745 6935 raft_consensus.cc:1261] T 4b10e180c11c4cd4a308fc7d311840c2 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.476033 7198 raft_consensus.cc:1261] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.476641 9331 consensus_queue.cc:1035] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.476998 8399 consensus_queue.cc:1035] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.481488 9331 raft_consensus.cc:2932] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:26.481657 7198 raft_consensus.cc:2932] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
W20240502 01:54:26.483300 7003 consensus_peers.cc:485] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 -> Peer 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198:37123): Couldn't send request to peer 9e47e5c098b848c9abe1559cbcbb6321. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b10e180c11c4cd4a308fc7d311840c2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:26.487691 6659 catalog_manager.cc:5484] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:26.492056 6935 raft_consensus.cc:2932] T 4b10e180c11c4cd4a308fc7d311840c2 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:26.495826 9506 ts_tablet_manager.cc:902] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321: Initiating tablet copy from peer 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:26.505677 9506 tablet_copy_client.cc:308] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.194:45315
I20240502 01:54:26.512187 6958 tablet_copy_service.cc:140] P 137e2183ff57411791e9af58dfab0a3c: Received BeginTabletCopySession request for tablet 01e98643312b4a2eb80e3bcb3d95674c from peer 9e47e5c098b848c9abe1559cbcbb6321 ({username='slave'} at 127.31.161.198:33415)
I20240502 01:54:26.512396 6958 tablet_copy_service.cc:161] P 137e2183ff57411791e9af58dfab0a3c: Beginning new tablet copy session on tablet 01e98643312b4a2eb80e3bcb3d95674c from peer 9e47e5c098b848c9abe1559cbcbb6321 at {username='slave'} at 127.31.161.198:33415: session id = 9e47e5c098b848c9abe1559cbcbb6321-01e98643312b4a2eb80e3bcb3d95674c
I20240502 01:54:26.514765 6958 tablet_copy_source_session.cc:205] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:26.516027 9506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e98643312b4a2eb80e3bcb3d95674c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.520906 9506 tablet_copy_client.cc:783] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:26.521385 9506 tablet_copy_client.cc:647] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:26.522325 6802 consensus_queue.cc:237] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:26.525367 9510 tablet_copy_client.cc:1002] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:26.525959 6935 raft_consensus.cc:1261] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.526151 7198 raft_consensus.cc:1261] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.527686 8246 consensus_queue.cc:1035] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.528566 8246 consensus_queue.cc:1035] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.530953 8246 raft_consensus.cc:2932] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:26.533650 6935 raft_consensus.cc:2932] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
W20240502 01:54:26.534857 6739 consensus_peers.cc:485] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd -> Peer 48866680637c40368e45ba844efb9298 (127.31.161.200:34863): Couldn't send request to peer 48866680637c40368e45ba844efb9298. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39f0a7df4f624eee81c25b41d19d1740. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:26.536170 7198 raft_consensus.cc:2932] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:26.539376 6659 catalog_manager.cc:5484] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c reported cstate change: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:26.556048 7067 consensus_queue.cc:237] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:26.559091 6802 raft_consensus.cc:1261] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.559757 7198 raft_consensus.cc:1261] T 46b6cd134f7e4b2fb56f4df44ed00f15 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.559787 9331 consensus_queue.cc:1035] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.560578 9331 consensus_queue.cc:1035] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:26.565963 7003 consensus_peers.cc:485] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 -> Peer 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197:44451): Couldn't send request to peer 3f156b7cfb4f40f481699b257153e5cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46b6cd134f7e4b2fb56f4df44ed00f15. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:26.574404 8399 raft_consensus.cc:2932] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:26.575057 7198 raft_consensus.cc:2932] T 46b6cd134f7e4b2fb56f4df44ed00f15 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:26.580262 6802 raft_consensus.cc:2932] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:26.580672 6653 catalog_manager.cc:5484] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:26.590464 9532 ts_tablet_manager.cc:902] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298: Initiating tablet copy from peer 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:26.597514 9532 tablet_copy_client.cc:308] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:46083
I20240502 01:54:26.604609 7089 tablet_copy_service.cc:140] P 11c097d7d32a4c22a6385a7465876eb3: Received BeginTabletCopySession request for tablet f6a165cbc26f4defb70866779f83bfdc from peer 48866680637c40368e45ba844efb9298 ({username='slave'} at 127.31.161.200:55399)
I20240502 01:54:26.604787 7089 tablet_copy_service.cc:161] P 11c097d7d32a4c22a6385a7465876eb3: Beginning new tablet copy session on tablet f6a165cbc26f4defb70866779f83bfdc from peer 48866680637c40368e45ba844efb9298 at {username='slave'} at 127.31.161.200:55399: session id = 48866680637c40368e45ba844efb9298-f6a165cbc26f4defb70866779f83bfdc
I20240502 01:54:26.607203 7089 tablet_copy_source_session.cc:205] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:26.608744 9532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a165cbc26f4defb70866779f83bfdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.612229 6802 consensus_queue.cc:237] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:26.614444 9532 tablet_copy_client.cc:783] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:26.615095 9532 tablet_copy_client.cc:647] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298: tablet copy: Starting download of 1 WAL segments...
W20240502 01:54:26.615250 6738 consensus_peers.cc:485] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd -> Peer c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199:35245): Couldn't send request to peer c39b0639d5fb4d9da09bf1d67d617fcc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1f4c94a105741339ba69e641244ac18. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:26.616024 6935 raft_consensus.cc:1261] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.616521 7067 raft_consensus.cc:1261] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.617168 9536 tablet_copy_client.cc:1002] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:26.617128 9288 consensus_queue.cc:1035] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.617408 8246 consensus_queue.cc:1035] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.620265 9288 raft_consensus.cc:2932] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:26.626334 6653 catalog_manager.cc:5484] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:26.626955 6935 raft_consensus.cc:2932] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:26.641384 7067 raft_consensus.cc:2932] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:26.663609 9541 ts_tablet_manager.cc:902] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc: Initiating tablet copy from peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:26.692531 9541 tablet_copy_client.cc:308] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.193:45489
I20240502 01:54:26.700646 7198 consensus_queue.cc:237] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:26.707685 6825 tablet_copy_service.cc:140] P 8214b6c4de6d4aa58a22af551eb8dedd: Received BeginTabletCopySession request for tablet f03b37fd2f844f61a39daadc698c9b29 from peer c39b0639d5fb4d9da09bf1d67d617fcc ({username='slave'} at 127.31.161.199:43359)
I20240502 01:54:26.707966 6825 tablet_copy_service.cc:161] P 8214b6c4de6d4aa58a22af551eb8dedd: Beginning new tablet copy session on tablet f03b37fd2f844f61a39daadc698c9b29 from peer c39b0639d5fb4d9da09bf1d67d617fcc at {username='slave'} at 127.31.161.199:43359: session id = c39b0639d5fb4d9da09bf1d67d617fcc-f03b37fd2f844f61a39daadc698c9b29
I20240502 01:54:26.711987 6825 tablet_copy_source_session.cc:205] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:26.714149 9541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f03b37fd2f844f61a39daadc698c9b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.718101 7067 raft_consensus.cc:1261] T 482f77b65a254ed3aec717c06698ffa7 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.718104 6935 raft_consensus.cc:1261] T 482f77b65a254ed3aec717c06698ffa7 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:26.718525 7134 consensus_peers.cc:485] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 -> Peer 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197:44451): Couldn't send request to peer 3f156b7cfb4f40f481699b257153e5cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 482f77b65a254ed3aec717c06698ffa7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:26.718879 9138 consensus_queue.cc:1035] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.719230 9484 consensus_queue.cc:1035] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.722177 9274 raft_consensus.cc:2932] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:26.722831 7067 raft_consensus.cc:2932] T 482f77b65a254ed3aec717c06698ffa7 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:26.724226 9541 tablet_copy_client.cc:783] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:26.724454 9541 tablet_copy_client.cc:647] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:26.726505 9552 tablet_copy_client.cc:1002] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:26.729939 6653 catalog_manager.cc:5484] T 482f77b65a254ed3aec717c06698ffa7 P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:26.734679 6935 raft_consensus.cc:2932] T 482f77b65a254ed3aec717c06698ffa7 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:26.762022 6802 consensus_queue.cc:237] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:26.765843 7198 raft_consensus.cc:1261] T a14c3c945b724df5ac3f917cef3186e0 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.766535 9288 consensus_queue.cc:1035] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.767158 6935 raft_consensus.cc:1261] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.771993 9288 consensus_queue.cc:1035] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:26.778021 6738 consensus_peers.cc:485] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd -> Peer 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198:37123): Couldn't send request to peer 9e47e5c098b848c9abe1559cbcbb6321. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a14c3c945b724df5ac3f917cef3186e0. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:26.781258 9288 raft_consensus.cc:2932] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:26.785413 7198 raft_consensus.cc:2932] T a14c3c945b724df5ac3f917cef3186e0 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:26.791112 6653 catalog_manager.cc:5484] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:26.796226 6935 raft_consensus.cc:2932] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:26.815059 6935 consensus_queue.cc:237] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:26.818032 7198 raft_consensus.cc:1261] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.818126 6802 raft_consensus.cc:1261] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.818795 9549 consensus_queue.cc:1035] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.819145 9395 consensus_queue.cc:1035] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:26.827986 6872 consensus_peers.cc:485] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c -> Peer 48866680637c40368e45ba844efb9298 (127.31.161.200:34863): Couldn't send request to peer 48866680637c40368e45ba844efb9298. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dea582c2912241609405695eed436abe. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:26.834836 9395 raft_consensus.cc:2932] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:26.837272 7198 raft_consensus.cc:2932] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:26.839649 6802 raft_consensus.cc:2932] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:26.843624 6654 catalog_manager.cc:5484] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:26.847641 9573 ts_tablet_manager.cc:902] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb: Initiating tablet copy from peer fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873)
I20240502 01:54:26.857237 9573 tablet_copy_client.cc:308] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:38873
I20240502 01:54:26.863348 7218 tablet_copy_service.cc:140] P fc21155d36d5416fa3650f0268a20b17: Received BeginTabletCopySession request for tablet d55da66aad2140d788704b056e476e99 from peer 3f156b7cfb4f40f481699b257153e5cb ({username='slave'} at 127.31.161.197:48453)
I20240502 01:54:26.863505 7218 tablet_copy_service.cc:161] P fc21155d36d5416fa3650f0268a20b17: Beginning new tablet copy session on tablet d55da66aad2140d788704b056e476e99 from peer 3f156b7cfb4f40f481699b257153e5cb at {username='slave'} at 127.31.161.197:48453: session id = 3f156b7cfb4f40f481699b257153e5cb-d55da66aad2140d788704b056e476e99
I20240502 01:54:26.866554 7218 tablet_copy_source_session.cc:205] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:26.868527 9573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55da66aad2140d788704b056e476e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.873766 6802 consensus_queue.cc:237] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:26.875105 9573 tablet_copy_client.cc:783] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:26.875340 9573 tablet_copy_client.cc:647] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:26.877962 7067 raft_consensus.cc:1261] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.878798 9516 consensus_queue.cc:1035] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.879693 9578 tablet_copy_client.cc:1002] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:26.879803 7198 raft_consensus.cc:1261] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.880402 9516 consensus_queue.cc:1035] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.883250 9516 raft_consensus.cc:2932] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
W20240502 01:54:26.884999 6738 consensus_peers.cc:485] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd -> Peer 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197:44451): Couldn't send request to peer 3f156b7cfb4f40f481699b257153e5cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 945224f5cff74edb9447bebc743e5086. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:26.886899 7198 raft_consensus.cc:2932] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:26.891724 6653 catalog_manager.cc:5484] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:26.896636 7067 raft_consensus.cc:2932] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:26.916342 6802 consensus_queue.cc:237] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:26.918947 7067 raft_consensus.cc:1261] T c303b823d03249f89bb6dfaca3934cc9 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:26.919075 6935 raft_consensus.cc:1261] T c303b823d03249f89bb6dfaca3934cc9 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:26.920007 6738 consensus_peers.cc:485] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd -> Peer c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199:35245): Couldn't send request to peer c39b0639d5fb4d9da09bf1d67d617fcc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c303b823d03249f89bb6dfaca3934cc9. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:26.920224 8246 consensus_queue.cc:1035] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.920763 9288 consensus_queue.cc:1035] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:26.934835 9288 raft_consensus.cc:2932] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:26.942301 6653 catalog_manager.cc:5484] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:26.944660 7067 raft_consensus.cc:2932] T c303b823d03249f89bb6dfaca3934cc9 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:26.946461 9596 ts_tablet_manager.cc:902] T ae02d485bc56444e994a56f759fbed1f P 48866680637c40368e45ba844efb9298: Initiating tablet copy from peer fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873)
I20240502 01:54:26.956604 9595 ts_tablet_manager.cc:902] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321: Initiating tablet copy from peer 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:26.960338 6935 raft_consensus.cc:2932] T c303b823d03249f89bb6dfaca3934cc9 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:26.966604 9596 tablet_copy_client.cc:308] T ae02d485bc56444e994a56f759fbed1f P 48866680637c40368e45ba844efb9298: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:38873
I20240502 01:54:26.967558 7218 tablet_copy_service.cc:140] P fc21155d36d5416fa3650f0268a20b17: Received BeginTabletCopySession request for tablet ae02d485bc56444e994a56f759fbed1f from peer 48866680637c40368e45ba844efb9298 ({username='slave'} at 127.31.161.200:55615)
I20240502 01:54:26.967777 7218 tablet_copy_service.cc:161] P fc21155d36d5416fa3650f0268a20b17: Beginning new tablet copy session on tablet ae02d485bc56444e994a56f759fbed1f from peer 48866680637c40368e45ba844efb9298 at {username='slave'} at 127.31.161.200:55615: session id = 48866680637c40368e45ba844efb9298-ae02d485bc56444e994a56f759fbed1f
I20240502 01:54:26.971199 7218 tablet_copy_source_session.cc:205] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:26.972892 9596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae02d485bc56444e994a56f759fbed1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:26.981753 9596 tablet_copy_client.cc:783] T ae02d485bc56444e994a56f759fbed1f P 48866680637c40368e45ba844efb9298: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:26.982029 9596 tablet_copy_client.cc:647] T ae02d485bc56444e994a56f759fbed1f P 48866680637c40368e45ba844efb9298: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:26.987437 9606 tablet_copy_client.cc:1002] T ae02d485bc56444e994a56f759fbed1f P 48866680637c40368e45ba844efb9298: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:26.990779 7198 consensus_queue.cc:237] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
W20240502 01:54:27.006140 7134 consensus_peers.cc:485] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 -> Peer 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198:37123): Couldn't send request to peer 9e47e5c098b848c9abe1559cbcbb6321. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d58f60c0920e4616850cb246acce1fd7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:27.007601 6802 raft_consensus.cc:1261] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:27.008383 9138 consensus_queue.cc:1035] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:27.012238 7067 raft_consensus.cc:1261] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:27.012982 9138 consensus_queue.cc:1035] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:27.017802 9484 raft_consensus.cc:2932] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:27.018463 7067 raft_consensus.cc:2932] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:27.031354 9609 ts_tablet_manager.cc:902] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc: Initiating tablet copy from peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:27.031088 6653 catalog_manager.cc:5484] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:27.035518 9595 tablet_copy_client.cc:308] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:46083
I20240502 01:54:27.037448 6802 raft_consensus.cc:2932] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:27.044478 7089 tablet_copy_service.cc:140] P 11c097d7d32a4c22a6385a7465876eb3: Received BeginTabletCopySession request for tablet 4b10e180c11c4cd4a308fc7d311840c2 from peer 9e47e5c098b848c9abe1559cbcbb6321 ({username='slave'} at 127.31.161.198:55919)
I20240502 01:54:27.044641 7089 tablet_copy_service.cc:161] P 11c097d7d32a4c22a6385a7465876eb3: Beginning new tablet copy session on tablet 4b10e180c11c4cd4a308fc7d311840c2 from peer 9e47e5c098b848c9abe1559cbcbb6321 at {username='slave'} at 127.31.161.198:55919: session id = 9e47e5c098b848c9abe1559cbcbb6321-4b10e180c11c4cd4a308fc7d311840c2
I20240502 01:54:27.047663 7089 tablet_copy_source_session.cc:205] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:27.049109 9595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b10e180c11c4cd4a308fc7d311840c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:27.057055 9595 tablet_copy_client.cc:783] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:27.057435 9595 tablet_copy_client.cc:647] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:27.059263 9614 tablet_copy_client.cc:1002] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:27.073622 9618 ts_tablet_manager.cc:902] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 3f156b7cfb4f40f481699b257153e5cb: Initiating tablet copy from peer 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:27.082952 9619 ts_tablet_manager.cc:902] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298: Initiating tablet copy from peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:27.084304 9618 tablet_copy_client.cc:308] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:46083
I20240502 01:54:27.095400 7089 tablet_copy_service.cc:140] P 11c097d7d32a4c22a6385a7465876eb3: Received BeginTabletCopySession request for tablet 46b6cd134f7e4b2fb56f4df44ed00f15 from peer 3f156b7cfb4f40f481699b257153e5cb ({username='slave'} at 127.31.161.197:34771)
I20240502 01:54:27.095592 7089 tablet_copy_service.cc:161] P 11c097d7d32a4c22a6385a7465876eb3: Beginning new tablet copy session on tablet 46b6cd134f7e4b2fb56f4df44ed00f15 from peer 3f156b7cfb4f40f481699b257153e5cb at {username='slave'} at 127.31.161.197:34771: session id = 3f156b7cfb4f40f481699b257153e5cb-46b6cd134f7e4b2fb56f4df44ed00f15
I20240502 01:54:27.099076 7089 tablet_copy_source_session.cc:205] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:27.100526 9618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46b6cd134f7e4b2fb56f4df44ed00f15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:27.107388 9609 tablet_copy_client.cc:308] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.193:45489
I20240502 01:54:27.107738 9618 tablet_copy_client.cc:783] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:27.108083 9618 tablet_copy_client.cc:647] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:27.108153 6825 tablet_copy_service.cc:140] P 8214b6c4de6d4aa58a22af551eb8dedd: Received BeginTabletCopySession request for tablet d1f4c94a105741339ba69e641244ac18 from peer c39b0639d5fb4d9da09bf1d67d617fcc ({username='slave'} at 127.31.161.199:43359)
I20240502 01:54:27.108357 6825 tablet_copy_service.cc:161] P 8214b6c4de6d4aa58a22af551eb8dedd: Beginning new tablet copy session on tablet d1f4c94a105741339ba69e641244ac18 from peer c39b0639d5fb4d9da09bf1d67d617fcc at {username='slave'} at 127.31.161.199:43359: session id = c39b0639d5fb4d9da09bf1d67d617fcc-d1f4c94a105741339ba69e641244ac18
I20240502 01:54:27.110044 9627 tablet_copy_client.cc:1002] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:27.111093 6825 tablet_copy_source_session.cc:205] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:27.112644 9609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f4c94a105741339ba69e641244ac18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:27.116547 9634 ts_tablet_manager.cc:902] T 482f77b65a254ed3aec717c06698ffa7 P 3f156b7cfb4f40f481699b257153e5cb: Initiating tablet copy from peer fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873)
I20240502 01:54:27.121702 9609 tablet_copy_client.cc:783] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:27.121941 9609 tablet_copy_client.cc:647] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:27.123452 9631 tablet_copy_client.cc:1002] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Injecting latency into file download: 1500
W20240502 01:54:27.134938 9518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:27.135370 9518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:54:27.139663 9634 tablet_copy_client.cc:308] T 482f77b65a254ed3aec717c06698ffa7 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:38873
I20240502 01:54:27.139833 9619 tablet_copy_client.cc:308] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.193:45489
I20240502 01:54:27.140466 7218 tablet_copy_service.cc:140] P fc21155d36d5416fa3650f0268a20b17: Received BeginTabletCopySession request for tablet 482f77b65a254ed3aec717c06698ffa7 from peer 3f156b7cfb4f40f481699b257153e5cb ({username='slave'} at 127.31.161.197:48453)
I20240502 01:54:27.140635 7218 tablet_copy_service.cc:161] P fc21155d36d5416fa3650f0268a20b17: Beginning new tablet copy session on tablet 482f77b65a254ed3aec717c06698ffa7 from peer 3f156b7cfb4f40f481699b257153e5cb at {username='slave'} at 127.31.161.197:48453: session id = 3f156b7cfb4f40f481699b257153e5cb-482f77b65a254ed3aec717c06698ffa7
I20240502 01:54:27.143648 7218 tablet_copy_source_session.cc:205] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:27.145797 9634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 482f77b65a254ed3aec717c06698ffa7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:54:27.147918 9518 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:54:27.171636 9634 tablet_copy_client.cc:783] T 482f77b65a254ed3aec717c06698ffa7 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:27.172034 9634 tablet_copy_client.cc:647] T 482f77b65a254ed3aec717c06698ffa7 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:27.173928 9644 tablet_copy_client.cc:1002] T 482f77b65a254ed3aec717c06698ffa7 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:27.186455 6825 tablet_copy_service.cc:140] P 8214b6c4de6d4aa58a22af551eb8dedd: Received BeginTabletCopySession request for tablet 39f0a7df4f624eee81c25b41d19d1740 from peer 48866680637c40368e45ba844efb9298 ({username='slave'} at 127.31.161.200:42825)
I20240502 01:54:27.186637 6825 tablet_copy_service.cc:161] P 8214b6c4de6d4aa58a22af551eb8dedd: Beginning new tablet copy session on tablet 39f0a7df4f624eee81c25b41d19d1740 from peer 48866680637c40368e45ba844efb9298 at {username='slave'} at 127.31.161.200:42825: session id = 48866680637c40368e45ba844efb9298-39f0a7df4f624eee81c25b41d19d1740
I20240502 01:54:27.190006 6825 tablet_copy_source_session.cc:205] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:27.191496 9619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f0a7df4f624eee81c25b41d19d1740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:27.199189 9655 ts_tablet_manager.cc:902] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321: Initiating tablet copy from peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:27.199452 9619 tablet_copy_client.cc:783] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:27.199730 9619 tablet_copy_client.cc:647] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:27.203197 9640 tablet_copy_client.cc:1002] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:27.227252 9655 tablet_copy_client.cc:308] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.193:45489
I20240502 01:54:27.238278 6825 tablet_copy_service.cc:140] P 8214b6c4de6d4aa58a22af551eb8dedd: Received BeginTabletCopySession request for tablet a14c3c945b724df5ac3f917cef3186e0 from peer 9e47e5c098b848c9abe1559cbcbb6321 ({username='slave'} at 127.31.161.198:60753)
I20240502 01:54:27.238467 6825 tablet_copy_service.cc:161] P 8214b6c4de6d4aa58a22af551eb8dedd: Beginning new tablet copy session on tablet a14c3c945b724df5ac3f917cef3186e0 from peer 9e47e5c098b848c9abe1559cbcbb6321 at {username='slave'} at 127.31.161.198:60753: session id = 9e47e5c098b848c9abe1559cbcbb6321-a14c3c945b724df5ac3f917cef3186e0
I20240502 01:54:27.241398 6825 tablet_copy_source_session.cc:205] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:27.243647 9655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a14c3c945b724df5ac3f917cef3186e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:27.250134 9655 tablet_copy_client.cc:783] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:27.250525 9655 tablet_copy_client.cc:647] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:27.252310 9666 tablet_copy_client.cc:1002] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:27.289552 9692 ts_tablet_manager.cc:902] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298: Initiating tablet copy from peer 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:27.291450 9692 tablet_copy_client.cc:308] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.194:45315
I20240502 01:54:27.299093 6958 tablet_copy_service.cc:140] P 137e2183ff57411791e9af58dfab0a3c: Received BeginTabletCopySession request for tablet dea582c2912241609405695eed436abe from peer 48866680637c40368e45ba844efb9298 ({username='slave'} at 127.31.161.200:55333)
I20240502 01:54:27.299253 6958 tablet_copy_service.cc:161] P 137e2183ff57411791e9af58dfab0a3c: Beginning new tablet copy session on tablet dea582c2912241609405695eed436abe from peer 48866680637c40368e45ba844efb9298 at {username='slave'} at 127.31.161.200:55333: session id = 48866680637c40368e45ba844efb9298-dea582c2912241609405695eed436abe
I20240502 01:54:27.302429 6958 tablet_copy_source_session.cc:205] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:27.303944 9692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea582c2912241609405695eed436abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:27.310979 9692 tablet_copy_client.cc:783] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:27.311318 9692 tablet_copy_client.cc:647] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:27.312963 9696 tablet_copy_client.cc:1002] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:27.454058 7047 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:27.454710 6915 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:27.456265 9743 ts_tablet_manager.cc:902] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc: Initiating tablet copy from peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:27.470654 8672 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:27.493384 6782 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:27.494505 9751 ts_tablet_manager.cc:902] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb: Initiating tablet copy from peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:27.514058 9743 tablet_copy_client.cc:308] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.193:45489
I20240502 01:54:27.520875 9067 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:27.525010 6825 tablet_copy_service.cc:140] P 8214b6c4de6d4aa58a22af551eb8dedd: Received BeginTabletCopySession request for tablet c303b823d03249f89bb6dfaca3934cc9 from peer c39b0639d5fb4d9da09bf1d67d617fcc ({username='slave'} at 127.31.161.199:43359)
I20240502 01:54:27.525209 6825 tablet_copy_service.cc:161] P 8214b6c4de6d4aa58a22af551eb8dedd: Beginning new tablet copy session on tablet c303b823d03249f89bb6dfaca3934cc9 from peer c39b0639d5fb4d9da09bf1d67d617fcc at {username='slave'} at 127.31.161.199:43359: session id = c39b0639d5fb4d9da09bf1d67d617fcc-c303b823d03249f89bb6dfaca3934cc9
I20240502 01:54:27.528264 6825 tablet_copy_source_session.cc:205] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:27.535207 9743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c303b823d03249f89bb6dfaca3934cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:27.551863 8804 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:27.563163 9743 tablet_copy_client.cc:783] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:27.563457 9743 tablet_copy_client.cc:647] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:27.566109 9756 tablet_copy_client.cc:1002] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:27.567440 7179 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 01:54:27.607702 8937 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:27.652035 9764 ts_tablet_manager.cc:902] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321: Initiating tablet copy from peer fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873)
I20240502 01:54:27.686980 9764 tablet_copy_client.cc:308] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:38873
I20240502 01:54:27.687709 7218 tablet_copy_service.cc:140] P fc21155d36d5416fa3650f0268a20b17: Received BeginTabletCopySession request for tablet d58f60c0920e4616850cb246acce1fd7 from peer 9e47e5c098b848c9abe1559cbcbb6321 ({username='slave'} at 127.31.161.198:52833)
I20240502 01:54:27.687904 7218 tablet_copy_service.cc:161] P fc21155d36d5416fa3650f0268a20b17: Beginning new tablet copy session on tablet d58f60c0920e4616850cb246acce1fd7 from peer 9e47e5c098b848c9abe1559cbcbb6321 at {username='slave'} at 127.31.161.198:52833: session id = 9e47e5c098b848c9abe1559cbcbb6321-d58f60c0920e4616850cb246acce1fd7
I20240502 01:54:27.690945 7218 tablet_copy_source_session.cc:205] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:27.692346 9764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58f60c0920e4616850cb246acce1fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:27.718389 9751 tablet_copy_client.cc:308] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.193:45489
I20240502 01:54:27.718967 9764 tablet_copy_client.cc:783] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:27.719234 9764 tablet_copy_client.cc:647] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:27.721904 9775 tablet_copy_client.cc:1002] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:27.735531 6825 tablet_copy_service.cc:140] P 8214b6c4de6d4aa58a22af551eb8dedd: Received BeginTabletCopySession request for tablet 945224f5cff74edb9447bebc743e5086 from peer 3f156b7cfb4f40f481699b257153e5cb ({username='slave'} at 127.31.161.197:46071)
I20240502 01:54:27.735741 6825 tablet_copy_service.cc:161] P 8214b6c4de6d4aa58a22af551eb8dedd: Beginning new tablet copy session on tablet 945224f5cff74edb9447bebc743e5086 from peer 3f156b7cfb4f40f481699b257153e5cb at {username='slave'} at 127.31.161.197:46071: session id = 3f156b7cfb4f40f481699b257153e5cb-945224f5cff74edb9447bebc743e5086
I20240502 01:54:27.738195 6825 tablet_copy_source_session.cc:205] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:27.739663 9751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 945224f5cff74edb9447bebc743e5086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:27.746186 9751 tablet_copy_client.cc:783] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:27.746603 9751 tablet_copy_client.cc:647] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:27.752983 9780 tablet_copy_client.cc:1002] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Injecting latency into file download: 1500
Runtime error: ksck discovered errors
I20240502 01:54:27.795028 9443 tablet_copy_client.cc:523] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:27.800194 9443 tablet_bootstrap.cc:492] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298: Bootstrap starting.
I20240502 01:54:27.811900 9443 log.cc:824] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:27.847460 9443 tablet_bootstrap.cc:492] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:27.848014 9443 tablet_bootstrap.cc:492] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298: Bootstrap complete.
I20240502 01:54:27.848423 9443 ts_tablet_manager.cc:1367] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298: Time spent bootstrapping tablet: real 0.049s user 0.040s sys 0.004s
I20240502 01:54:27.853790 9443 raft_consensus.cc:348] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:27.854159 9443 raft_consensus.cc:729] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 48866680637c40368e45ba844efb9298, State: Initialized, Role: LEARNER
I20240502 01:54:27.854621 9443 consensus_queue.cc:260] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:27.856211 9443 ts_tablet_manager.cc:1398] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298: Time spent starting tablet: real 0.008s user 0.009s sys 0.001s
I20240502 01:54:27.857231 7218 tablet_copy_service.cc:342] P fc21155d36d5416fa3650f0268a20b17: Request end of tablet copy session 48866680637c40368e45ba844efb9298-7d8c4a55e09d41e391b8ba318dfe6896 received from {username='slave'} at 127.31.161.200:55615
I20240502 01:54:27.857410 7218 tablet_copy_service.cc:433] P fc21155d36d5416fa3650f0268a20b17: ending tablet copy session 48866680637c40368e45ba844efb9298-7d8c4a55e09d41e391b8ba318dfe6896 on tablet 7d8c4a55e09d41e391b8ba318dfe6896 with peer 48866680637c40368e45ba844efb9298
I20240502 01:54:27.878675 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.201:0
--local_ip_for_outbound_sockets=127.31.161.201
--webserver_interface=127.31.161.201
--webserver_port=0
--tserver_master_addrs=127.31.161.254:36997
--builtin_ntp_servers=127.31.161.212:43969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 01:54:27.885828 9446 tablet_copy_client.cc:523] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:27.890125 9446 tablet_bootstrap.cc:492] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap starting.
I20240502 01:54:27.901028 9446 log.cc:824] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:27.914793 9453 tablet_copy_client.cc:523] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:27.919250 9455 tablet_copy_client.cc:523] T dabfafc485c144b995aa4248fb1e897b P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:27.919749 9453 tablet_bootstrap.cc:492] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap starting.
I20240502 01:54:27.922688 9476 tablet_copy_client.cc:523] T c3397a96d989416fa2bccdc71c7ea9ec P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:27.924067 9455 tablet_bootstrap.cc:492] T dabfafc485c144b995aa4248fb1e897b P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap starting.
I20240502 01:54:27.925909 9476 tablet_bootstrap.cc:492] T c3397a96d989416fa2bccdc71c7ea9ec P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap starting.
I20240502 01:54:27.935778 9453 log.cc:824] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:27.948959 9455 log.cc:824] T dabfafc485c144b995aa4248fb1e897b P c39b0639d5fb4d9da09bf1d67d617fcc: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:27.996142 9446 tablet_bootstrap.cc:492] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:27.996764 9446 tablet_bootstrap.cc:492] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap complete.
I20240502 01:54:27.997267 9446 ts_tablet_manager.cc:1367] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb: Time spent bootstrapping tablet: real 0.107s user 0.053s sys 0.004s
I20240502 01:54:28.014626 9446 raft_consensus.cc:348] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:28.015172 9446 raft_consensus.cc:729] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3f156b7cfb4f40f481699b257153e5cb, State: Initialized, Role: LEARNER
I20240502 01:54:28.015612 9476 tablet_bootstrap.cc:492] T c3397a96d989416fa2bccdc71c7ea9ec P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:28.015868 9446 consensus_queue.cc:260] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:28.016309 9476 tablet_bootstrap.cc:492] T c3397a96d989416fa2bccdc71c7ea9ec P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap complete.
I20240502 01:54:28.016804 9476 ts_tablet_manager.cc:1367] T c3397a96d989416fa2bccdc71c7ea9ec P c39b0639d5fb4d9da09bf1d67d617fcc: Time spent bootstrapping tablet: real 0.091s user 0.053s sys 0.007s
I20240502 01:54:28.023147 9476 raft_consensus.cc:348] T c3397a96d989416fa2bccdc71c7ea9ec P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:28.024303 9476 raft_consensus.cc:729] T c3397a96d989416fa2bccdc71c7ea9ec P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c39b0639d5fb4d9da09bf1d67d617fcc, State: Initialized, Role: LEARNER
I20240502 01:54:28.024894 9476 consensus_queue.cc:260] T c3397a96d989416fa2bccdc71c7ea9ec P c39b0639d5fb4d9da09bf1d67d617fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:28.025717 9506 tablet_copy_client.cc:523] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:28.025983 9453 tablet_bootstrap.cc:492] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:28.026553 9453 tablet_bootstrap.cc:492] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap complete.
I20240502 01:54:28.028409 9453 ts_tablet_manager.cc:1367] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321: Time spent bootstrapping tablet: real 0.109s user 0.050s sys 0.033s
I20240502 01:54:28.029376 9506 tablet_bootstrap.cc:492] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap starting.
I20240502 01:54:28.034747 9446 ts_tablet_manager.cc:1398] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb: Time spent starting tablet: real 0.037s user 0.006s sys 0.002s
I20240502 01:54:28.036104 6958 tablet_copy_service.cc:342] P 137e2183ff57411791e9af58dfab0a3c: Request end of tablet copy session 3f156b7cfb4f40f481699b257153e5cb-db7a48f131f141f0a4e45036434f8129 received from {username='slave'} at 127.31.161.197:41393
I20240502 01:54:28.036298 6958 tablet_copy_service.cc:433] P 137e2183ff57411791e9af58dfab0a3c: ending tablet copy session 3f156b7cfb4f40f481699b257153e5cb-db7a48f131f141f0a4e45036434f8129 on tablet db7a48f131f141f0a4e45036434f8129 with peer 3f156b7cfb4f40f481699b257153e5cb
I20240502 01:54:28.040750 9453 raft_consensus.cc:348] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:28.041600 9455 tablet_bootstrap.cc:492] T dabfafc485c144b995aa4248fb1e897b P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:28.042115 9455 tablet_bootstrap.cc:492] T dabfafc485c144b995aa4248fb1e897b P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap complete.
I20240502 01:54:28.042433 9455 ts_tablet_manager.cc:1367] T dabfafc485c144b995aa4248fb1e897b P c39b0639d5fb4d9da09bf1d67d617fcc: Time spent bootstrapping tablet: real 0.119s user 0.044s sys 0.007s
I20240502 01:54:28.043509 9455 raft_consensus.cc:348] T dabfafc485c144b995aa4248fb1e897b P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:28.043756 9455 raft_consensus.cc:729] T dabfafc485c144b995aa4248fb1e897b P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c39b0639d5fb4d9da09bf1d67d617fcc, State: Initialized, Role: LEARNER
I20240502 01:54:28.043991 9455 consensus_queue.cc:260] T dabfafc485c144b995aa4248fb1e897b P c39b0639d5fb4d9da09bf1d67d617fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:28.041283 9453 raft_consensus.cc:729] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e47e5c098b848c9abe1559cbcbb6321, State: Initialized, Role: LEARNER
I20240502 01:54:28.056247 9453 consensus_queue.cc:260] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:28.056910 8957 raft_consensus.cc:1203] T dabfafc485c144b995aa4248fb1e897b P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:28.058045 9455 ts_tablet_manager.cc:1398] T dabfafc485c144b995aa4248fb1e897b P c39b0639d5fb4d9da09bf1d67d617fcc: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20240502 01:54:28.058936 7218 tablet_copy_service.cc:342] P fc21155d36d5416fa3650f0268a20b17: Request end of tablet copy session c39b0639d5fb4d9da09bf1d67d617fcc-dabfafc485c144b995aa4248fb1e897b received from {username='slave'} at 127.31.161.199:57741
I20240502 01:54:28.059087 7218 tablet_copy_service.cc:433] P fc21155d36d5416fa3650f0268a20b17: ending tablet copy session c39b0639d5fb4d9da09bf1d67d617fcc-dabfafc485c144b995aa4248fb1e897b on tablet dabfafc485c144b995aa4248fb1e897b with peer c39b0639d5fb4d9da09bf1d67d617fcc
I20240502 01:54:28.062508 9476 ts_tablet_manager.cc:1398] T c3397a96d989416fa2bccdc71c7ea9ec P c39b0639d5fb4d9da09bf1d67d617fcc: Time spent starting tablet: real 0.045s user 0.008s sys 0.006s
I20240502 01:54:28.063308 7089 tablet_copy_service.cc:342] P 11c097d7d32a4c22a6385a7465876eb3: Request end of tablet copy session c39b0639d5fb4d9da09bf1d67d617fcc-c3397a96d989416fa2bccdc71c7ea9ec received from {username='slave'} at 127.31.161.199:34537
I20240502 01:54:28.063462 7089 tablet_copy_service.cc:433] P 11c097d7d32a4c22a6385a7465876eb3: ending tablet copy session c39b0639d5fb4d9da09bf1d67d617fcc-c3397a96d989416fa2bccdc71c7ea9ec on tablet c3397a96d989416fa2bccdc71c7ea9ec with peer c39b0639d5fb4d9da09bf1d67d617fcc
I20240502 01:54:28.067812 9453 ts_tablet_manager.cc:1398] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321: Time spent starting tablet: real 0.039s user 0.008s sys 0.003s
I20240502 01:54:28.075251 7218 tablet_copy_service.cc:342] P fc21155d36d5416fa3650f0268a20b17: Request end of tablet copy session 9e47e5c098b848c9abe1559cbcbb6321-5acc7c2405464a328b2016bb2293f7e8 received from {username='slave'} at 127.31.161.198:52833
I20240502 01:54:28.075392 7218 tablet_copy_service.cc:433] P fc21155d36d5416fa3650f0268a20b17: ending tablet copy session 9e47e5c098b848c9abe1559cbcbb6321-5acc7c2405464a328b2016bb2293f7e8 on tablet 5acc7c2405464a328b2016bb2293f7e8 with peer 9e47e5c098b848c9abe1559cbcbb6321
I20240502 01:54:28.102483 9506 tablet_bootstrap.cc:492] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:28.102970 9506 tablet_bootstrap.cc:492] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap complete.
I20240502 01:54:28.103291 9506 ts_tablet_manager.cc:1367] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321: Time spent bootstrapping tablet: real 0.074s user 0.043s sys 0.004s
I20240502 01:54:28.104310 9506 raft_consensus.cc:348] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:28.104593 9506 raft_consensus.cc:729] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e47e5c098b848c9abe1559cbcbb6321, State: Initialized, Role: LEARNER
I20240502 01:54:28.104806 9506 consensus_queue.cc:260] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:28.105422 9506 ts_tablet_manager.cc:1398] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:54:28.106247 6958 tablet_copy_service.cc:342] P 137e2183ff57411791e9af58dfab0a3c: Request end of tablet copy session 9e47e5c098b848c9abe1559cbcbb6321-01e98643312b4a2eb80e3bcb3d95674c received from {username='slave'} at 127.31.161.198:33415
I20240502 01:54:28.106428 6958 tablet_copy_service.cc:433] P 137e2183ff57411791e9af58dfab0a3c: ending tablet copy session 9e47e5c098b848c9abe1559cbcbb6321-01e98643312b4a2eb80e3bcb3d95674c on tablet 01e98643312b4a2eb80e3bcb3d95674c with peer 9e47e5c098b848c9abe1559cbcbb6321
I20240502 01:54:28.117522 9532 tablet_copy_client.cc:523] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:28.120127 9532 tablet_bootstrap.cc:492] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298: Bootstrap starting.
I20240502 01:54:28.145417 8692 raft_consensus.cc:1203] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:28.154666 8823 raft_consensus.cc:1203] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:28.168087 9532 tablet_bootstrap.cc:492] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:28.168457 9532 tablet_bootstrap.cc:492] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298: Bootstrap complete.
I20240502 01:54:28.168725 9532 ts_tablet_manager.cc:1367] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298: Time spent bootstrapping tablet: real 0.049s user 0.035s sys 0.004s
I20240502 01:54:28.169734 9532 raft_consensus.cc:348] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:28.169984 9532 raft_consensus.cc:729] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 48866680637c40368e45ba844efb9298, State: Initialized, Role: LEARNER
I20240502 01:54:28.170217 9532 consensus_queue.cc:260] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:28.172358 9532 ts_tablet_manager.cc:1398] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 01:54:28.174499 7089 tablet_copy_service.cc:342] P 11c097d7d32a4c22a6385a7465876eb3: Request end of tablet copy session 48866680637c40368e45ba844efb9298-f6a165cbc26f4defb70866779f83bfdc received from {username='slave'} at 127.31.161.200:55399
I20240502 01:54:28.174624 7089 tablet_copy_service.cc:433] P 11c097d7d32a4c22a6385a7465876eb3: ending tablet copy session 48866680637c40368e45ba844efb9298-f6a165cbc26f4defb70866779f83bfdc on tablet f6a165cbc26f4defb70866779f83bfdc with peer 48866680637c40368e45ba844efb9298
I20240502 01:54:28.193794 8957 raft_consensus.cc:1203] T c3397a96d989416fa2bccdc71c7ea9ec P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:28.226919 9541 tablet_copy_client.cc:523] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:28.229804 9541 tablet_bootstrap.cc:492] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap starting.
I20240502 01:54:28.269940 9087 raft_consensus.cc:1203] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:28.289355 9541 tablet_bootstrap.cc:492] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:28.289985 9541 tablet_bootstrap.cc:492] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap complete.
I20240502 01:54:28.290302 9541 ts_tablet_manager.cc:1367] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc: Time spent bootstrapping tablet: real 0.061s user 0.035s sys 0.016s
I20240502 01:54:28.291095 9541 raft_consensus.cc:348] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:28.291376 9541 raft_consensus.cc:729] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c39b0639d5fb4d9da09bf1d67d617fcc, State: Initialized, Role: LEARNER
I20240502 01:54:28.291612 9541 consensus_queue.cc:260] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:28.293401 9541 ts_tablet_manager.cc:1398] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:28.294273 6825 tablet_copy_service.cc:342] P 8214b6c4de6d4aa58a22af551eb8dedd: Request end of tablet copy session c39b0639d5fb4d9da09bf1d67d617fcc-f03b37fd2f844f61a39daadc698c9b29 received from {username='slave'} at 127.31.161.199:43359
I20240502 01:54:28.294436 6825 tablet_copy_service.cc:433] P 8214b6c4de6d4aa58a22af551eb8dedd: ending tablet copy session c39b0639d5fb4d9da09bf1d67d617fcc-f03b37fd2f844f61a39daadc698c9b29 on tablet f03b37fd2f844f61a39daadc698c9b29 with peer c39b0639d5fb4d9da09bf1d67d617fcc
W20240502 01:54:28.337704 9811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:28.338110 9811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:28.338289 9811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:28.347874 9811 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:28.348045 9811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:28.348250 9811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.201
I20240502 01:54:28.360160 9811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:43969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.201:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.31.161.201
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.31.161.254:36997
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.201
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:28.361840 9811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:28.362165 8823 raft_consensus.cc:1203] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:28.364156 9811 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 01:54:28.380132 9573 tablet_copy_client.cc:523] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 01:54:28.381676 9896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:28.382769 9811 server_base.cc:1034] running on GCE node
I20240502 01:54:28.383436 9573 tablet_bootstrap.cc:492] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap starting.
W20240502 01:54:28.383582 9895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:28.390323 9898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:28.391135 9811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:28.392645 9811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:28.394665 9811 hybrid_clock.cc:648] HybridClock initialized: now 1714614868394609 us; error 51 us; skew 500 ppm
I20240502 01:54:28.395056 9811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:28.408118 9811 webserver.cc:460] Webserver started at http://127.31.161.201:41841/ using document root <none> and password file <none>
I20240502 01:54:28.409298 9811 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:28.409451 9811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:28.409855 9811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:28.413621 9811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data/instance:
uuid: "fdfc60a29ea9465c88ee45c3c332caa1"
format_stamp: "Formatted at 2024-05-02 01:54:28 on dist-test-slave-06l8"
I20240502 01:54:28.414605 9811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/wal/instance:
uuid: "fdfc60a29ea9465c88ee45c3c332caa1"
format_stamp: "Formatted at 2024-05-02 01:54:28 on dist-test-slave-06l8"
I20240502 01:54:28.421969 9811 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20240502 01:54:28.430703 9573 tablet_bootstrap.cc:492] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:28.431174 9573 tablet_bootstrap.cc:492] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap complete.
I20240502 01:54:28.431497 9573 ts_tablet_manager.cc:1367] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb: Time spent bootstrapping tablet: real 0.048s user 0.041s sys 0.005s
I20240502 01:54:28.432575 9573 raft_consensus.cc:348] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:28.432883 9573 raft_consensus.cc:729] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3f156b7cfb4f40f481699b257153e5cb, State: Initialized, Role: LEARNER
I20240502 01:54:28.433145 9573 consensus_queue.cc:260] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:28.433743 9573 ts_tablet_manager.cc:1398] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:54:28.434791 7218 tablet_copy_service.cc:342] P fc21155d36d5416fa3650f0268a20b17: Request end of tablet copy session 3f156b7cfb4f40f481699b257153e5cb-d55da66aad2140d788704b056e476e99 received from {username='slave'} at 127.31.161.197:48453
I20240502 01:54:28.434937 7218 tablet_copy_service.cc:433] P fc21155d36d5416fa3650f0268a20b17: ending tablet copy session 3f156b7cfb4f40f481699b257153e5cb-d55da66aad2140d788704b056e476e99 on tablet d55da66aad2140d788704b056e476e99 with peer 3f156b7cfb4f40f481699b257153e5cb
I20240502 01:54:28.436117 9920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:28.438311 9811 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.012s sys 0.001s
I20240502 01:54:28.438495 9811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "fdfc60a29ea9465c88ee45c3c332caa1"
format_stamp: "Formatted at 2024-05-02 01:54:28 on dist-test-slave-06l8"
I20240502 01:54:28.438755 9811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:28.459676 9811 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:28.460975 9811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:28.461402 9811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:28.462594 9811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:28.464236 9811 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:28.464329 9811 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:28.464438 9811 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:28.464521 9811 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:28.487902 9596 tablet_copy_client.cc:523] T ae02d485bc56444e994a56f759fbed1f P 48866680637c40368e45ba844efb9298: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:28.491756 9596 tablet_bootstrap.cc:492] T ae02d485bc56444e994a56f759fbed1f P 48866680637c40368e45ba844efb9298: Bootstrap starting.
I20240502 01:54:28.516079 9811 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.201:43211
I20240502 01:54:28.516093 10053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.201:43211 every 8 connection(s)
I20240502 01:54:28.517891 9811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240502 01:54:28.522897 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 9811
I20240502 01:54:28.523133 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/wal/instance
I20240502 01:54:28.531890 9596 tablet_bootstrap.cc:492] T ae02d485bc56444e994a56f759fbed1f P 48866680637c40368e45ba844efb9298: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:28.532336 9596 tablet_bootstrap.cc:492] T ae02d485bc56444e994a56f759fbed1f P 48866680637c40368e45ba844efb9298: Bootstrap complete.
I20240502 01:54:28.532614 9596 ts_tablet_manager.cc:1367] T ae02d485bc56444e994a56f759fbed1f P 48866680637c40368e45ba844efb9298: Time spent bootstrapping tablet: real 0.041s user 0.040s sys 0.000s
I20240502 01:54:28.532774 9484 raft_consensus.cc:1050] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17: attempting to promote NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc to VOTER
I20240502 01:54:28.533457 9596 raft_consensus.cc:348] T ae02d485bc56444e994a56f759fbed1f P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:28.533732 9596 raft_consensus.cc:729] T ae02d485bc56444e994a56f759fbed1f P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 48866680637c40368e45ba844efb9298, State: Initialized, Role: LEARNER
I20240502 01:54:28.533838 9484 consensus_queue.cc:237] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:28.533936 9596 consensus_queue.cc:260] T ae02d485bc56444e994a56f759fbed1f P 48866680637c40368e45ba844efb9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:28.534590 9596 ts_tablet_manager.cc:1398] T ae02d485bc56444e994a56f759fbed1f P 48866680637c40368e45ba844efb9298: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:54:28.535965 7218 tablet_copy_service.cc:342] P fc21155d36d5416fa3650f0268a20b17: Request end of tablet copy session 48866680637c40368e45ba844efb9298-ae02d485bc56444e994a56f759fbed1f received from {username='slave'} at 127.31.161.200:55615
I20240502 01:54:28.536098 7218 tablet_copy_service.cc:433] P fc21155d36d5416fa3650f0268a20b17: ending tablet copy session 48866680637c40368e45ba844efb9298-ae02d485bc56444e994a56f759fbed1f on tablet ae02d485bc56444e994a56f759fbed1f with peer 48866680637c40368e45ba844efb9298
I20240502 01:54:28.536522 10054 heartbeater.cc:347] Connected to a master server at 127.31.161.254:36997
I20240502 01:54:28.536855 10054 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:28.538019 10054 heartbeater.cc:511] Master 127.31.161.254:36997 requested a full tablet report, sending...
I20240502 01:54:28.540192 8957 raft_consensus.cc:1261] T dabfafc485c144b995aa4248fb1e897b P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:28.540287 6657 ts_manager.cc:194] Registered new tserver with Master: fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201:43211)
I20240502 01:54:28.540864 9274 consensus_queue.cc:1035] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:28.541257 6657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.201:55931
I20240502 01:54:28.542443 7067 raft_consensus.cc:1261] T dabfafc485c144b995aa4248fb1e897b P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:28.543219 9138 consensus_queue.cc:1035] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:28.546069 9274 raft_consensus.cc:2932] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:28.546798 7067 raft_consensus.cc:2932] T dabfafc485c144b995aa4248fb1e897b P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:28.552687 6657 catalog_manager.cc:5484] T dabfafc485c144b995aa4248fb1e897b P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:28.556489 6802 raft_consensus.cc:1261] T dabfafc485c144b995aa4248fb1e897b P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:28.556385 8957 raft_consensus.cc:2932] T dabfafc485c144b995aa4248fb1e897b P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:28.559216 9138 consensus_queue.cc:1035] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:28.559630 9595 tablet_copy_client.cc:523] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:28.563043 9595 tablet_bootstrap.cc:492] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap starting.
I20240502 01:54:28.568655 6803 raft_consensus.cc:2932] T dabfafc485c144b995aa4248fb1e897b P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:28.576146 7198 consensus_queue.cc:237] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:28.578284 8957 raft_consensus.cc:1261] T dabfafc485c144b995aa4248fb1e897b P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:28.579142 7067 raft_consensus.cc:1261] T dabfafc485c144b995aa4248fb1e897b P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:28.579213 9138 consensus_queue.cc:1035] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:28.579878 10059 consensus_queue.cc:1035] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240502 01:54:28.582777 9274 raft_consensus.cc:2932] T dabfafc485c144b995aa4248fb1e897b P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:28.583325 8957 raft_consensus.cc:2932] T dabfafc485c144b995aa4248fb1e897b P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:28.584020 7067 raft_consensus.cc:2932] T dabfafc485c144b995aa4248fb1e897b P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:28.588007 9879 raft_consensus.cc:1050] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c: attempting to promote NON_VOTER 3f156b7cfb4f40f481699b257153e5cb to VOTER
I20240502 01:54:28.588907 9879 consensus_queue.cc:237] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:28.589672 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dabfafc485c144b995aa4248fb1e897b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:28.589927 6654 catalog_manager.cc:5484] T dabfafc485c144b995aa4248fb1e897b P c39b0639d5fb4d9da09bf1d67d617fcc reported cstate change: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:28.593986 8692 raft_consensus.cc:1261] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:28.594372 7198 raft_consensus.cc:1261] T db7a48f131f141f0a4e45036434f8129 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:28.594965 6782 tablet_service.cc:1514] Processing DeleteTablet for tablet dabfafc485c144b995aa4248fb1e897b with delete_type TABLET_DATA_TOMBSTONED (TS 8214b6c4de6d4aa58a22af551eb8dedd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35388
I20240502 01:54:28.595047 9878 consensus_queue.cc:1035] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:28.595945 10070 tablet_replica.cc:330] T dabfafc485c144b995aa4248fb1e897b P 8214b6c4de6d4aa58a22af551eb8dedd: stopping tablet replica
I20240502 01:54:28.594949 6803 raft_consensus.cc:1261] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:28.596503 10070 raft_consensus.cc:2229] T dabfafc485c144b995aa4248fb1e897b P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:28.596800 10070 raft_consensus.cc:2258] T dabfafc485c144b995aa4248fb1e897b P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:28.597213 9395 consensus_queue.cc:1035] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:28.597805 9395 consensus_queue.cc:1035] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:28.598408 10070 ts_tablet_manager.cc:1875] T dabfafc485c144b995aa4248fb1e897b P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:28.601991 9395 raft_consensus.cc:2932] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:28.605902 8692 raft_consensus.cc:2932] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:28.606657 10070 ts_tablet_manager.cc:1888] T dabfafc485c144b995aa4248fb1e897b P 8214b6c4de6d4aa58a22af551eb8dedd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:28.606851 10070 log.cc:1196] T dabfafc485c144b995aa4248fb1e897b P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/dabfafc485c144b995aa4248fb1e897b
I20240502 01:54:28.609506 6645 catalog_manager.cc:4830] TS 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): tablet dabfafc485c144b995aa4248fb1e897b (table rebalance_test_table_6 [id=981b993f808c4320845154760fb7959f]) successfully deleted
I20240502 01:54:28.610386 9618 tablet_copy_client.cc:523] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:28.610939 6654 catalog_manager.cc:5484] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb reported cstate change: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:28.613967 9618 tablet_bootstrap.cc:492] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap starting.
I20240502 01:54:28.623842 9609 tablet_copy_client.cc:523] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:28.627084 9595 tablet_bootstrap.cc:492] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:28.627573 9595 tablet_bootstrap.cc:492] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap complete.
I20240502 01:54:28.627832 9609 tablet_bootstrap.cc:492] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap starting.
I20240502 01:54:28.627876 9595 ts_tablet_manager.cc:1367] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321: Time spent bootstrapping tablet: real 0.065s user 0.037s sys 0.004s
I20240502 01:54:28.628703 9595 raft_consensus.cc:348] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:28.628957 9595 raft_consensus.cc:729] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e47e5c098b848c9abe1559cbcbb6321, State: Initialized, Role: LEARNER
I20240502 01:54:28.629460 9595 consensus_queue.cc:260] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:28.630196 9595 ts_tablet_manager.cc:1398] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:54:28.631337 7089 tablet_copy_service.cc:342] P 11c097d7d32a4c22a6385a7465876eb3: Request end of tablet copy session 9e47e5c098b848c9abe1559cbcbb6321-4b10e180c11c4cd4a308fc7d311840c2 received from {username='slave'} at 127.31.161.198:55919
I20240502 01:54:28.631492 7089 tablet_copy_service.cc:433] P 11c097d7d32a4c22a6385a7465876eb3: ending tablet copy session 9e47e5c098b848c9abe1559cbcbb6321-4b10e180c11c4cd4a308fc7d311840c2 on tablet 4b10e180c11c4cd4a308fc7d311840c2 with peer 9e47e5c098b848c9abe1559cbcbb6321
I20240502 01:54:28.642640 7198 raft_consensus.cc:2932] T db7a48f131f141f0a4e45036434f8129 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:28.648007 6803 raft_consensus.cc:2932] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:28.656438 6935 consensus_queue.cc:237] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:28.659317 8692 raft_consensus.cc:1261] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:28.660391 6803 raft_consensus.cc:1261] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:28.660845 10074 consensus_queue.cc:1035] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:28.661698 9395 consensus_queue.cc:1035] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:28.664040 9879 raft_consensus.cc:2932] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:28.666291 8692 raft_consensus.cc:2932] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:28.672838 6654 catalog_manager.cc:5484] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb reported cstate change: config changed from index 5 to 6, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:28.674968 9634 tablet_copy_client.cc:523] T 482f77b65a254ed3aec717c06698ffa7 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:28.678932 8957 raft_consensus.cc:1203] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:28.679770 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet db7a48f131f141f0a4e45036434f8129 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:28.680758 7179 tablet_service.cc:1514] Processing DeleteTablet for tablet db7a48f131f141f0a4e45036434f8129 with delete_type TABLET_DATA_TOMBSTONED (TS fc21155d36d5416fa3650f0268a20b17 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40840
I20240502 01:54:28.681447 6803 raft_consensus.cc:2932] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:28.687453 9087 raft_consensus.cc:1203] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:28.687669 10078 tablet_replica.cc:330] T db7a48f131f141f0a4e45036434f8129 P fc21155d36d5416fa3650f0268a20b17: stopping tablet replica
I20240502 01:54:28.688181 10078 raft_consensus.cc:2229] T db7a48f131f141f0a4e45036434f8129 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:28.693022 9634 tablet_bootstrap.cc:492] T 482f77b65a254ed3aec717c06698ffa7 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap starting.
I20240502 01:54:28.697872 10078 raft_consensus.cc:2258] T db7a48f131f141f0a4e45036434f8129 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:28.700219 10078 ts_tablet_manager.cc:1875] T db7a48f131f141f0a4e45036434f8129 P fc21155d36d5416fa3650f0268a20b17: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:28.703585 9619 tablet_copy_client.cc:523] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:28.707118 9619 tablet_bootstrap.cc:492] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298: Bootstrap starting.
I20240502 01:54:28.714701 10078 ts_tablet_manager.cc:1888] T db7a48f131f141f0a4e45036434f8129 P fc21155d36d5416fa3650f0268a20b17: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:28.715781 10078 log.cc:1196] T db7a48f131f141f0a4e45036434f8129 P fc21155d36d5416fa3650f0268a20b17: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/db7a48f131f141f0a4e45036434f8129
I20240502 01:54:28.716856 6645 catalog_manager.cc:4830] TS fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873): tablet db7a48f131f141f0a4e45036434f8129 (table rebalance_test_table_6 [id=981b993f808c4320845154760fb7959f]) successfully deleted
I20240502 01:54:28.726217 9618 tablet_bootstrap.cc:492] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:28.726629 9618 tablet_bootstrap.cc:492] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap complete.
I20240502 01:54:28.726924 9618 ts_tablet_manager.cc:1367] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 3f156b7cfb4f40f481699b257153e5cb: Time spent bootstrapping tablet: real 0.113s user 0.046s sys 0.008s
I20240502 01:54:28.727780 9618 raft_consensus.cc:348] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:28.728071 9618 raft_consensus.cc:729] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3f156b7cfb4f40f481699b257153e5cb, State: Initialized, Role: LEARNER
I20240502 01:54:28.728303 9618 consensus_queue.cc:260] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 3f156b7cfb4f40f481699b257153e5cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:28.728982 9618 ts_tablet_manager.cc:1398] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 3f156b7cfb4f40f481699b257153e5cb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:54:28.729779 7089 tablet_copy_service.cc:342] P 11c097d7d32a4c22a6385a7465876eb3: Request end of tablet copy session 3f156b7cfb4f40f481699b257153e5cb-46b6cd134f7e4b2fb56f4df44ed00f15 received from {username='slave'} at 127.31.161.197:34771
I20240502 01:54:28.729914 7089 tablet_copy_service.cc:433] P 11c097d7d32a4c22a6385a7465876eb3: ending tablet copy session 3f156b7cfb4f40f481699b257153e5cb-46b6cd134f7e4b2fb56f4df44ed00f15 on tablet 46b6cd134f7e4b2fb56f4df44ed00f15 with peer 3f156b7cfb4f40f481699b257153e5cb
I20240502 01:54:28.737843 9609 tablet_bootstrap.cc:492] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:28.738371 9609 tablet_bootstrap.cc:492] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap complete.
I20240502 01:54:28.738739 9609 ts_tablet_manager.cc:1367] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc: Time spent bootstrapping tablet: real 0.111s user 0.067s sys 0.003s
I20240502 01:54:28.739610 9609 raft_consensus.cc:348] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:28.740039 9609 raft_consensus.cc:729] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c39b0639d5fb4d9da09bf1d67d617fcc, State: Initialized, Role: LEARNER
I20240502 01:54:28.740312 9609 consensus_queue.cc:260] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:28.741086 9609 ts_tablet_manager.cc:1398] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:28.741853 6825 tablet_copy_service.cc:342] P 8214b6c4de6d4aa58a22af551eb8dedd: Request end of tablet copy session c39b0639d5fb4d9da09bf1d67d617fcc-d1f4c94a105741339ba69e641244ac18 received from {username='slave'} at 127.31.161.199:43359
I20240502 01:54:28.741976 6825 tablet_copy_service.cc:433] P 8214b6c4de6d4aa58a22af551eb8dedd: ending tablet copy session c39b0639d5fb4d9da09bf1d67d617fcc-d1f4c94a105741339ba69e641244ac18 on tablet d1f4c94a105741339ba69e641244ac18 with peer c39b0639d5fb4d9da09bf1d67d617fcc
I20240502 01:54:28.752698 9655 tablet_copy_client.cc:523] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:28.756321 9655 tablet_bootstrap.cc:492] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap starting.
I20240502 01:54:28.763231 10064 raft_consensus.cc:1050] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17: attempting to promote NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 to VOTER
I20240502 01:54:28.764060 10064 consensus_queue.cc:237] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:28.771338 7067 raft_consensus.cc:1261] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:28.771358 6935 raft_consensus.cc:1261] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:28.772466 8823 raft_consensus.cc:1261] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:28.772611 9274 consensus_queue.cc:1035] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:28.773201 10059 consensus_queue.cc:1035] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:28.774092 8692 raft_consensus.cc:1203] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:28.774873 9274 consensus_queue.cc:1035] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:28.776824 9634 tablet_bootstrap.cc:492] T 482f77b65a254ed3aec717c06698ffa7 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:28.777318 9634 tablet_bootstrap.cc:492] T 482f77b65a254ed3aec717c06698ffa7 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap complete.
I20240502 01:54:28.777588 9634 ts_tablet_manager.cc:1367] T 482f77b65a254ed3aec717c06698ffa7 P 3f156b7cfb4f40f481699b257153e5cb: Time spent bootstrapping tablet: real 0.085s user 0.045s sys 0.006s
I20240502 01:54:28.778474 9634 raft_consensus.cc:348] T 482f77b65a254ed3aec717c06698ffa7 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:28.778723 9634 raft_consensus.cc:729] T 482f77b65a254ed3aec717c06698ffa7 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3f156b7cfb4f40f481699b257153e5cb, State: Initialized, Role: LEARNER
I20240502 01:54:28.778966 9634 consensus_queue.cc:260] T 482f77b65a254ed3aec717c06698ffa7 P 3f156b7cfb4f40f481699b257153e5cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:28.780278 9634 ts_tablet_manager.cc:1398] T 482f77b65a254ed3aec717c06698ffa7 P 3f156b7cfb4f40f481699b257153e5cb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:28.781746 7218 tablet_copy_service.cc:342] P fc21155d36d5416fa3650f0268a20b17: Request end of tablet copy session 3f156b7cfb4f40f481699b257153e5cb-482f77b65a254ed3aec717c06698ffa7 received from {username='slave'} at 127.31.161.197:48453
I20240502 01:54:28.781872 7218 tablet_copy_service.cc:433] P fc21155d36d5416fa3650f0268a20b17: ending tablet copy session 3f156b7cfb4f40f481699b257153e5cb-482f77b65a254ed3aec717c06698ffa7 on tablet 482f77b65a254ed3aec717c06698ffa7 with peer 3f156b7cfb4f40f481699b257153e5cb
I20240502 01:54:28.782368 9619 tablet_bootstrap.cc:492] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:28.783221 9619 tablet_bootstrap.cc:492] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298: Bootstrap complete.
I20240502 01:54:28.783860 9619 ts_tablet_manager.cc:1367] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298: Time spent bootstrapping tablet: real 0.077s user 0.046s sys 0.005s
I20240502 01:54:28.784634 9619 raft_consensus.cc:348] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:28.784895 9619 raft_consensus.cc:729] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 48866680637c40368e45ba844efb9298, State: Initialized, Role: LEARNER
I20240502 01:54:28.785153 9619 consensus_queue.cc:260] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:28.786685 9619 ts_tablet_manager.cc:1398] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:28.787346 6825 tablet_copy_service.cc:342] P 8214b6c4de6d4aa58a22af551eb8dedd: Request end of tablet copy session 48866680637c40368e45ba844efb9298-39f0a7df4f624eee81c25b41d19d1740 received from {username='slave'} at 127.31.161.200:42825
I20240502 01:54:28.787475 6825 tablet_copy_service.cc:433] P 8214b6c4de6d4aa58a22af551eb8dedd: ending tablet copy session 48866680637c40368e45ba844efb9298-39f0a7df4f624eee81c25b41d19d1740 on tablet 39f0a7df4f624eee81c25b41d19d1740 with peer 48866680637c40368e45ba844efb9298
I20240502 01:54:28.791707 9274 raft_consensus.cc:2932] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:28.795491 6935 raft_consensus.cc:2932] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:28.795720 7067 raft_consensus.cc:2932] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:28.801016 6654 catalog_manager.cc:5484] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:28.800647 8823 raft_consensus.cc:2932] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:28.805975 7198 consensus_queue.cc:237] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:28.809401 6936 raft_consensus.cc:1261] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:28.810441 9274 consensus_queue.cc:1035] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:28.812889 9138 raft_consensus.cc:2932] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:28.816190 9692 tablet_copy_client.cc:523] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:28.819700 9692 tablet_bootstrap.cc:492] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298: Bootstrap starting.
I20240502 01:54:28.822758 8824 raft_consensus.cc:1261] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:28.824393 8399 raft_consensus.cc:1050] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3: attempting to promote NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc to VOTER
I20240502 01:54:28.825174 9484 consensus_queue.cc:1035] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:28.825894 8399 consensus_queue.cc:237] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:28.826828 8824 raft_consensus.cc:2932] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:28.829998 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5acc7c2405464a328b2016bb2293f7e8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:28.832626 6654 catalog_manager.cc:5484] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:28.834949 6935 raft_consensus.cc:2932] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:28.848027 9484 raft_consensus.cc:1050] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17: attempting to promote NON_VOTER 48866680637c40368e45ba844efb9298 to VOTER
I20240502 01:54:28.848865 9484 consensus_queue.cc:237] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:28.850353 7047 tablet_service.cc:1514] Processing DeleteTablet for tablet 5acc7c2405464a328b2016bb2293f7e8 with delete_type TABLET_DATA_TOMBSTONED (TS 11c097d7d32a4c22a6385a7465876eb3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59762
I20240502 01:54:28.853935 9655 tablet_bootstrap.cc:492] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:28.854449 9655 tablet_bootstrap.cc:492] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap complete.
I20240502 01:54:28.854565 6803 raft_consensus.cc:1261] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:28.854732 9655 ts_tablet_manager.cc:1367] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321: Time spent bootstrapping tablet: real 0.098s user 0.043s sys 0.001s
I20240502 01:54:28.855588 9655 raft_consensus.cc:348] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:28.855814 9087 raft_consensus.cc:1261] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:28.855871 9655 raft_consensus.cc:729] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e47e5c098b848c9abe1559cbcbb6321, State: Initialized, Role: LEARNER
I20240502 01:54:28.856103 9655 consensus_queue.cc:260] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:28.856693 9274 consensus_queue.cc:1035] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:28.856880 8957 raft_consensus.cc:1261] T c3397a96d989416fa2bccdc71c7ea9ec P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:28.857285 9655 ts_tablet_manager.cc:1398] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:54:28.858125 6825 tablet_copy_service.cc:342] P 8214b6c4de6d4aa58a22af551eb8dedd: Request end of tablet copy session 9e47e5c098b848c9abe1559cbcbb6321-a14c3c945b724df5ac3f917cef3186e0 received from {username='slave'} at 127.31.161.198:60753
I20240502 01:54:28.858115 9274 consensus_queue.cc:1035] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:28.858268 6825 tablet_copy_service.cc:433] P 8214b6c4de6d4aa58a22af551eb8dedd: ending tablet copy session 9e47e5c098b848c9abe1559cbcbb6321-a14c3c945b724df5ac3f917cef3186e0 on tablet a14c3c945b724df5ac3f917cef3186e0 with peer 9e47e5c098b848c9abe1559cbcbb6321
I20240502 01:54:28.853201 10104 tablet_replica.cc:330] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3: stopping tablet replica
I20240502 01:54:28.857518 10079 consensus_queue.cc:1035] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:28.858763 6935 raft_consensus.cc:1261] T c3397a96d989416fa2bccdc71c7ea9ec P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:28.860648 10079 consensus_queue.cc:1035] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:28.861546 6935 raft_consensus.cc:1261] T 7d8c4a55e09d41e391b8ba318dfe6896 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:28.862110 9484 consensus_queue.cc:1035] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:28.864539 6802 raft_consensus.cc:1261] T c3397a96d989416fa2bccdc71c7ea9ec P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:28.865334 10079 consensus_queue.cc:1035] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:28.865610 10104 raft_consensus.cc:2229] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:28.866858 10104 raft_consensus.cc:2258] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:28.871749 10104 ts_tablet_manager.cc:1875] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:28.881615 9484 raft_consensus.cc:2932] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:28.883378 10079 raft_consensus.cc:2932] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:28.888065 10104 ts_tablet_manager.cc:1888] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:28.887786 6804 raft_consensus.cc:2932] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:28.888252 10104 log.cc:1196] T 5acc7c2405464a328b2016bb2293f7e8 P 11c097d7d32a4c22a6385a7465876eb3: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/wals/5acc7c2405464a328b2016bb2293f7e8
I20240502 01:54:28.888598 9087 raft_consensus.cc:2932] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:28.890658 6654 catalog_manager.cc:5484] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:28.891047 6935 raft_consensus.cc:2932] T 7d8c4a55e09d41e391b8ba318dfe6896 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:28.898996 7198 consensus_queue.cc:237] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:28.901510 9086 raft_consensus.cc:1261] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:28.902177 9274 consensus_queue.cc:1035] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:28.903380 6803 raft_consensus.cc:1261] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:28.904441 9484 consensus_queue.cc:1035] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:28.906723 9138 raft_consensus.cc:2932] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:28.907284 6803 raft_consensus.cc:2932] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:28.910609 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d8c4a55e09d41e391b8ba318dfe6896 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:28.911008 6801 raft_consensus.cc:2932] T c3397a96d989416fa2bccdc71c7ea9ec P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:28.912021 9087 raft_consensus.cc:2932] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:28.913447 6658 catalog_manager.cc:5484] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:28.916877 8692 raft_consensus.cc:1203] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:28.921885 6657 catalog_manager.cc:5484] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:28.926621 9086 raft_consensus.cc:1203] T ae02d485bc56444e994a56f759fbed1f P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:28.933333 6915 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d8c4a55e09d41e391b8ba318dfe6896 with delete_type TABLET_DATA_TOMBSTONED (TS 137e2183ff57411791e9af58dfab0a3c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58498
W20240502 01:54:28.943545 7003 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:45315, user_credentials={real_user=slave}} blocked reactor thread for 49990.7us
I20240502 01:54:28.944671 8957 raft_consensus.cc:2932] T c3397a96d989416fa2bccdc71c7ea9ec P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:28.950747 10115 tablet_replica.cc:330] T 7d8c4a55e09d41e391b8ba318dfe6896 P 137e2183ff57411791e9af58dfab0a3c: stopping tablet replica
I20240502 01:54:28.951512 10115 raft_consensus.cc:2229] T 7d8c4a55e09d41e391b8ba318dfe6896 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:28.952419 8824 raft_consensus.cc:1203] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:28.952862 6646 catalog_manager.cc:4830] TS 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083): tablet 5acc7c2405464a328b2016bb2293f7e8 (table rebalance_test_table_1 [id=9a08c31578ae4224915289c7c289ee7a]) successfully deleted
I20240502 01:54:28.955466 7067 consensus_queue.cc:237] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:28.958438 8957 raft_consensus.cc:1261] T c3397a96d989416fa2bccdc71c7ea9ec P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:28.959371 10112 consensus_queue.cc:1035] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:28.961916 9692 tablet_bootstrap.cc:492] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:28.962291 9692 tablet_bootstrap.cc:492] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298: Bootstrap complete.
I20240502 01:54:28.962544 9692 ts_tablet_manager.cc:1367] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298: Time spent bootstrapping tablet: real 0.143s user 0.050s sys 0.009s
I20240502 01:54:28.962893 10113 raft_consensus.cc:2932] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:28.963380 9692 raft_consensus.cc:348] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:28.963640 9692 raft_consensus.cc:729] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 48866680637c40368e45ba844efb9298, State: Initialized, Role: LEARNER
I20240502 01:54:28.963433 8957 raft_consensus.cc:2932] T c3397a96d989416fa2bccdc71c7ea9ec P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:28.963932 9692 consensus_queue.cc:260] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:28.964601 9692 ts_tablet_manager.cc:1398] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:54:28.967561 10115 raft_consensus.cc:2258] T 7d8c4a55e09d41e391b8ba318dfe6896 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:28.968288 6654 catalog_manager.cc:5484] T c3397a96d989416fa2bccdc71c7ea9ec P c39b0639d5fb4d9da09bf1d67d617fcc reported cstate change: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:28.969844 6935 raft_consensus.cc:2932] T c3397a96d989416fa2bccdc71c7ea9ec P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:28.972735 6782 tablet_service.cc:1514] Processing DeleteTablet for tablet c3397a96d989416fa2bccdc71c7ea9ec with delete_type TABLET_DATA_TOMBSTONED (TS 8214b6c4de6d4aa58a22af551eb8dedd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35388
I20240502 01:54:28.973490 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c3397a96d989416fa2bccdc71c7ea9ec with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:28.973944 10070 tablet_replica.cc:330] T c3397a96d989416fa2bccdc71c7ea9ec P 8214b6c4de6d4aa58a22af551eb8dedd: stopping tablet replica
I20240502 01:54:28.974121 6936 raft_consensus.cc:1261] T c3397a96d989416fa2bccdc71c7ea9ec P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:28.974157 10070 raft_consensus.cc:2229] T c3397a96d989416fa2bccdc71c7ea9ec P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:28.974470 6958 tablet_copy_service.cc:342] P 137e2183ff57411791e9af58dfab0a3c: Request end of tablet copy session 48866680637c40368e45ba844efb9298-dea582c2912241609405695eed436abe received from {username='slave'} at 127.31.161.200:55333
I20240502 01:54:28.974507 10070 raft_consensus.cc:2258] T c3397a96d989416fa2bccdc71c7ea9ec P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:28.974601 6958 tablet_copy_service.cc:433] P 137e2183ff57411791e9af58dfab0a3c: ending tablet copy session 48866680637c40368e45ba844efb9298-dea582c2912241609405695eed436abe on tablet dea582c2912241609405695eed436abe with peer 48866680637c40368e45ba844efb9298
I20240502 01:54:28.975093 10115 ts_tablet_manager.cc:1875] T 7d8c4a55e09d41e391b8ba318dfe6896 P 137e2183ff57411791e9af58dfab0a3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:28.978057 8399 consensus_queue.cc:1035] T c3397a96d989416fa2bccdc71c7ea9ec P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:28.979866 6936 raft_consensus.cc:2932] T c3397a96d989416fa2bccdc71c7ea9ec P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:28.981377 10070 ts_tablet_manager.cc:1875] T c3397a96d989416fa2bccdc71c7ea9ec P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:28.987201 9395 raft_consensus.cc:1050] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c: attempting to promote NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 to VOTER
I20240502 01:54:28.988004 9395 consensus_queue.cc:237] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:28.989176 10115 ts_tablet_manager.cc:1888] T 7d8c4a55e09d41e391b8ba318dfe6896 P 137e2183ff57411791e9af58dfab0a3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:28.989346 10115 log.cc:1196] T 7d8c4a55e09d41e391b8ba318dfe6896 P 137e2183ff57411791e9af58dfab0a3c: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/wals/7d8c4a55e09d41e391b8ba318dfe6896
I20240502 01:54:28.990459 6646 catalog_manager.cc:4830] TS 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315): tablet 7d8c4a55e09d41e391b8ba318dfe6896 (table rebalance_test_table_8 [id=8abdf44bc9e2414b9e4f80bf4c87c953]) successfully deleted
I20240502 01:54:28.991600 10070 ts_tablet_manager.cc:1888] T c3397a96d989416fa2bccdc71c7ea9ec P 8214b6c4de6d4aa58a22af551eb8dedd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:28.991767 10070 log.cc:1196] T c3397a96d989416fa2bccdc71c7ea9ec P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/c3397a96d989416fa2bccdc71c7ea9ec
I20240502 01:54:28.992002 7067 raft_consensus.cc:1261] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:28.992269 8824 raft_consensus.cc:1261] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:28.992565 9878 consensus_queue.cc:1035] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:28.992766 6645 catalog_manager.cc:4830] TS 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): tablet c3397a96d989416fa2bccdc71c7ea9ec (table rebalance_test_table_4 [id=9231a7576c804559bc4a4c54496d5223]) successfully deleted
I20240502 01:54:28.992828 9879 consensus_queue.cc:1035] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:28.993913 7198 raft_consensus.cc:1261] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:28.995136 9395 consensus_queue.cc:1035] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.009750 9878 raft_consensus.cc:2932] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.013449 7067 raft_consensus.cc:2932] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.016006 8824 raft_consensus.cc:2932] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.017464 8823 raft_consensus.cc:1203] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:29.021739 6657 catalog_manager.cc:5484] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c reported cstate change: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:29.032888 7198 raft_consensus.cc:2932] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.037243 6936 consensus_queue.cc:237] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:29.039383 8824 raft_consensus.cc:1261] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.039714 7198 raft_consensus.cc:1261] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.040241 9878 consensus_queue.cc:1035] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:29.040498 10074 consensus_queue.cc:1035] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:29.042937 9395 raft_consensus.cc:2932] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.046015 8824 raft_consensus.cc:2932] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.046828 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 01e98643312b4a2eb80e3bcb3d95674c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:29.049357 7199 raft_consensus.cc:2932] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.048976 6657 catalog_manager.cc:5484] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c reported cstate change: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:29.059659 7047 tablet_service.cc:1514] Processing DeleteTablet for tablet 01e98643312b4a2eb80e3bcb3d95674c with delete_type TABLET_DATA_TOMBSTONED (TS 11c097d7d32a4c22a6385a7465876eb3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59762
I20240502 01:54:29.060017 10104 tablet_replica.cc:330] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3: stopping tablet replica
I20240502 01:54:29.060266 10104 raft_consensus.cc:2229] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:29.060609 10104 raft_consensus.cc:2258] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:29.061626 10104 ts_tablet_manager.cc:1875] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:29.066550 9743 tablet_copy_client.cc:523] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:29.067425 10104 ts_tablet_manager.cc:1888] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:29.067528 10104 log.cc:1196] T 01e98643312b4a2eb80e3bcb3d95674c P 11c097d7d32a4c22a6385a7465876eb3: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/wals/01e98643312b4a2eb80e3bcb3d95674c
I20240502 01:54:29.068367 6646 catalog_manager.cc:4830] TS 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083): tablet 01e98643312b4a2eb80e3bcb3d95674c (table rebalance_test_table_3 [id=76b4b12b25de439c840462959d467432]) successfully deleted
I20240502 01:54:29.069610 9743 tablet_bootstrap.cc:492] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap starting.
I20240502 01:54:29.111752 9743 tablet_bootstrap.cc:492] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:29.112309 9743 tablet_bootstrap.cc:492] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap complete.
I20240502 01:54:29.112600 9743 ts_tablet_manager.cc:1367] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc: Time spent bootstrapping tablet: real 0.043s user 0.039s sys 0.001s
I20240502 01:54:29.113332 9743 raft_consensus.cc:348] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:29.113616 9743 raft_consensus.cc:729] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c39b0639d5fb4d9da09bf1d67d617fcc, State: Initialized, Role: LEARNER
I20240502 01:54:29.113824 9743 consensus_queue.cc:260] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:29.114390 9743 ts_tablet_manager.cc:1398] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:29.115732 6825 tablet_copy_service.cc:342] P 8214b6c4de6d4aa58a22af551eb8dedd: Request end of tablet copy session c39b0639d5fb4d9da09bf1d67d617fcc-c303b823d03249f89bb6dfaca3934cc9 received from {username='slave'} at 127.31.161.199:43359
I20240502 01:54:29.115862 6825 tablet_copy_service.cc:433] P 8214b6c4de6d4aa58a22af551eb8dedd: ending tablet copy session c39b0639d5fb4d9da09bf1d67d617fcc-c303b823d03249f89bb6dfaca3934cc9 on tablet c303b823d03249f89bb6dfaca3934cc9 with peer c39b0639d5fb4d9da09bf1d67d617fcc
I20240502 01:54:29.116032 8957 raft_consensus.cc:1203] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:29.121543 10112 raft_consensus.cc:1050] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3: attempting to promote NON_VOTER 48866680637c40368e45ba844efb9298 to VOTER
I20240502 01:54:29.122342 10112 consensus_queue.cc:237] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:29.124579 6936 raft_consensus.cc:1261] T f6a165cbc26f4defb70866779f83bfdc P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.124629 6803 raft_consensus.cc:1261] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.125171 10112 consensus_queue.cc:1035] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.125363 9087 raft_consensus.cc:1261] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.125746 10114 consensus_queue.cc:1035] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.126070 8399 consensus_queue.cc:1035] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.129186 10114 raft_consensus.cc:2932] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.130883 6936 raft_consensus.cc:2932] T f6a165cbc26f4defb70866779f83bfdc P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.132696 9087 raft_consensus.cc:2932] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.135308 6654 catalog_manager.cc:5484] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:29.136878 6803 raft_consensus.cc:2932] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.144694 7067 consensus_queue.cc:237] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:29.146602 6803 raft_consensus.cc:1261] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.147099 9087 raft_consensus.cc:1261] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.147639 10112 consensus_queue.cc:1035] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:29.148192 10112 consensus_queue.cc:1035] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:29.149958 10114 raft_consensus.cc:2932] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.150934 6803 raft_consensus.cc:2932] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.151257 9087 raft_consensus.cc:2932] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.154497 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f6a165cbc26f4defb70866779f83bfdc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:29.158495 6657 catalog_manager.cc:5484] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:29.162619 6915 tablet_service.cc:1514] Processing DeleteTablet for tablet f6a165cbc26f4defb70866779f83bfdc with delete_type TABLET_DATA_TOMBSTONED (TS 137e2183ff57411791e9af58dfab0a3c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58498
I20240502 01:54:29.163230 10115 tablet_replica.cc:330] T f6a165cbc26f4defb70866779f83bfdc P 137e2183ff57411791e9af58dfab0a3c: stopping tablet replica
I20240502 01:54:29.163436 10115 raft_consensus.cc:2229] T f6a165cbc26f4defb70866779f83bfdc P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:29.163789 10115 raft_consensus.cc:2258] T f6a165cbc26f4defb70866779f83bfdc P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:29.165283 10115 ts_tablet_manager.cc:1875] T f6a165cbc26f4defb70866779f83bfdc P 137e2183ff57411791e9af58dfab0a3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:29.171236 9087 raft_consensus.cc:1203] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:29.171612 8692 raft_consensus.cc:1203] T 482f77b65a254ed3aec717c06698ffa7 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:29.180702 10115 ts_tablet_manager.cc:1888] T f6a165cbc26f4defb70866779f83bfdc P 137e2183ff57411791e9af58dfab0a3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:29.180832 10115 log.cc:1196] T f6a165cbc26f4defb70866779f83bfdc P 137e2183ff57411791e9af58dfab0a3c: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/wals/f6a165cbc26f4defb70866779f83bfdc
I20240502 01:54:29.181581 6646 catalog_manager.cc:4830] TS 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315): tablet f6a165cbc26f4defb70866779f83bfdc (table rebalance_test_table_3 [id=76b4b12b25de439c840462959d467432]) successfully deleted
I20240502 01:54:29.203028 9516 raft_consensus.cc:1050] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd: attempting to promote NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc to VOTER
I20240502 01:54:29.204107 9516 consensus_queue.cc:237] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:29.210347 6936 raft_consensus.cc:1261] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.210906 8957 raft_consensus.cc:1261] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.211742 8246 consensus_queue.cc:1035] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.212329 8246 consensus_queue.cc:1035] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.217244 7199 raft_consensus.cc:1261] T f03b37fd2f844f61a39daadc698c9b29 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.217892 9516 consensus_queue.cc:1035] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.222486 9516 raft_consensus.cc:2932] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:29.222959 8957 raft_consensus.cc:2932] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:29.225780 9764 tablet_copy_client.cc:523] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:29.227164 7199 raft_consensus.cc:2932] T f03b37fd2f844f61a39daadc698c9b29 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:29.228608 6659 catalog_manager.cc:5484] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc reported cstate change: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:29.230126 9764 tablet_bootstrap.cc:492] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap starting.
I20240502 01:54:29.238593 6935 raft_consensus.cc:2932] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:29.250393 6803 consensus_queue.cc:237] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:29.252614 8957 raft_consensus.cc:1261] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.253232 8246 consensus_queue.cc:1035] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:29.253662 9751 tablet_copy_client.cc:523] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:29.254381 6935 raft_consensus.cc:1261] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.254935 8246 consensus_queue.cc:1035] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:29.255627 9516 raft_consensus.cc:2932] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:29.256362 8957 raft_consensus.cc:2932] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:29.257035 9751 tablet_bootstrap.cc:492] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap starting.
I20240502 01:54:29.259500 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f03b37fd2f844f61a39daadc698c9b29 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:29.261514 6935 raft_consensus.cc:2932] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:29.262096 6659 catalog_manager.cc:5484] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc reported cstate change: config changed from index 5 to 6, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:29.266425 7179 tablet_service.cc:1514] Processing DeleteTablet for tablet f03b37fd2f844f61a39daadc698c9b29 with delete_type TABLET_DATA_TOMBSTONED (TS fc21155d36d5416fa3650f0268a20b17 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40840
I20240502 01:54:29.275003 10193 tablet_replica.cc:330] T f03b37fd2f844f61a39daadc698c9b29 P fc21155d36d5416fa3650f0268a20b17: stopping tablet replica
I20240502 01:54:29.275228 10193 raft_consensus.cc:2229] T f03b37fd2f844f61a39daadc698c9b29 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:29.275610 10193 raft_consensus.cc:2258] T f03b37fd2f844f61a39daadc698c9b29 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:29.277318 10193 ts_tablet_manager.cc:1875] T f03b37fd2f844f61a39daadc698c9b29 P fc21155d36d5416fa3650f0268a20b17: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:29.303318 9764 tablet_bootstrap.cc:492] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:29.303766 9764 tablet_bootstrap.cc:492] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap complete.
I20240502 01:54:29.304054 9764 ts_tablet_manager.cc:1367] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321: Time spent bootstrapping tablet: real 0.074s user 0.038s sys 0.010s
I20240502 01:54:29.304886 9764 raft_consensus.cc:348] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:29.305172 9764 raft_consensus.cc:729] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e47e5c098b848c9abe1559cbcbb6321, State: Initialized, Role: LEARNER
I20240502 01:54:29.305359 9764 consensus_queue.cc:260] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:29.306674 10193 ts_tablet_manager.cc:1888] T f03b37fd2f844f61a39daadc698c9b29 P fc21155d36d5416fa3650f0268a20b17: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:29.306862 10193 log.cc:1196] T f03b37fd2f844f61a39daadc698c9b29 P fc21155d36d5416fa3650f0268a20b17: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/f03b37fd2f844f61a39daadc698c9b29
I20240502 01:54:29.307682 6645 catalog_manager.cc:4830] TS fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873): tablet f03b37fd2f844f61a39daadc698c9b29 (table rebalance_test_table_2 [id=3316a56a8d6043c6b05160d6e55a33e0]) successfully deleted
I20240502 01:54:29.308120 9764 ts_tablet_manager.cc:1398] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 01:54:29.310925 7218 tablet_copy_service.cc:342] P fc21155d36d5416fa3650f0268a20b17: Request end of tablet copy session 9e47e5c098b848c9abe1559cbcbb6321-d58f60c0920e4616850cb246acce1fd7 received from {username='slave'} at 127.31.161.198:52833
I20240502 01:54:29.311039 7218 tablet_copy_service.cc:433] P fc21155d36d5416fa3650f0268a20b17: ending tablet copy session 9e47e5c098b848c9abe1559cbcbb6321-d58f60c0920e4616850cb246acce1fd7 on tablet d58f60c0920e4616850cb246acce1fd7 with peer 9e47e5c098b848c9abe1559cbcbb6321
I20240502 01:54:29.327687 9751 tablet_bootstrap.cc:492] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:29.328194 9751 tablet_bootstrap.cc:492] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap complete.
I20240502 01:54:29.328512 9751 ts_tablet_manager.cc:1367] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb: Time spent bootstrapping tablet: real 0.072s user 0.037s sys 0.006s
I20240502 01:54:29.329524 9751 raft_consensus.cc:348] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:29.329957 9751 raft_consensus.cc:729] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3f156b7cfb4f40f481699b257153e5cb, State: Initialized, Role: LEARNER
I20240502 01:54:29.330184 9751 consensus_queue.cc:260] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:29.330900 9751 ts_tablet_manager.cc:1398] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb: Time spent starting tablet: real 0.002s user 0.005s sys 0.001s
I20240502 01:54:29.332222 6825 tablet_copy_service.cc:342] P 8214b6c4de6d4aa58a22af551eb8dedd: Request end of tablet copy session 3f156b7cfb4f40f481699b257153e5cb-945224f5cff74edb9447bebc743e5086 received from {username='slave'} at 127.31.161.197:46071
I20240502 01:54:29.332363 6825 tablet_copy_service.cc:433] P 8214b6c4de6d4aa58a22af551eb8dedd: ending tablet copy session 3f156b7cfb4f40f481699b257153e5cb-945224f5cff74edb9447bebc743e5086 on tablet 945224f5cff74edb9447bebc743e5086 with peer 3f156b7cfb4f40f481699b257153e5cb
I20240502 01:54:29.350968 10113 raft_consensus.cc:1050] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3: attempting to promote NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 to VOTER
I20240502 01:54:29.351926 10113 consensus_queue.cc:237] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:29.354598 8824 raft_consensus.cc:1261] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.355075 6935 raft_consensus.cc:1261] T 4b10e180c11c4cd4a308fc7d311840c2 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.355247 10099 consensus_queue.cc:1035] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.355809 7199 raft_consensus.cc:1261] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.356346 10099 consensus_queue.cc:1035] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.356875 10099 consensus_queue.cc:1035] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.360670 10112 raft_consensus.cc:1050] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3: attempting to promote NON_VOTER 3f156b7cfb4f40f481699b257153e5cb to VOTER
I20240502 01:54:29.361440 10112 consensus_queue.cc:237] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:29.363268 8399 raft_consensus.cc:2932] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.363951 6935 raft_consensus.cc:2932] T 4b10e180c11c4cd4a308fc7d311840c2 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.363986 7199 raft_consensus.cc:2932] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.366375 7198 raft_consensus.cc:1261] T 46b6cd134f7e4b2fb56f4df44ed00f15 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.366803 6803 raft_consensus.cc:1261] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.366933 10099 consensus_queue.cc:1035] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.368043 10114 consensus_queue.cc:1035] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.372117 6653 catalog_manager.cc:5484] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.373378 8692 raft_consensus.cc:1261] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.373873 10112 consensus_queue.cc:1035] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.375690 8824 raft_consensus.cc:2932] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.376253 10114 raft_consensus.cc:2932] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.377027 8692 raft_consensus.cc:2932] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.382328 6653 catalog_manager.cc:5484] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 3f156b7cfb4f40f481699b257153e5cb reported cstate change: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.384196 7067 consensus_queue.cc:237] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:29.384501 7199 raft_consensus.cc:2932] T 46b6cd134f7e4b2fb56f4df44ed00f15 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.386209 8824 raft_consensus.cc:1261] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.386587 7198 raft_consensus.cc:1261] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.386708 10113 consensus_queue.cc:1035] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:29.387275 10079 consensus_queue.cc:1035] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:29.389358 10079 raft_consensus.cc:2932] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.390473 6801 raft_consensus.cc:2932] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.391599 7067 consensus_queue.cc:237] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:29.393638 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4b10e180c11c4cd4a308fc7d311840c2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:29.394531 8823 raft_consensus.cc:2932] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.397883 8692 raft_consensus.cc:1261] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.398737 10099 consensus_queue.cc:1035] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:29.398717 7198 raft_consensus.cc:2932] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.401821 10079 raft_consensus.cc:2932] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.402459 6659 catalog_manager.cc:5484] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321 reported cstate change: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.402441 8692 raft_consensus.cc:2932] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.406392 7199 raft_consensus.cc:1261] T 46b6cd134f7e4b2fb56f4df44ed00f15 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.406399 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 46b6cd134f7e4b2fb56f4df44ed00f15 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:29.407092 6657 catalog_manager.cc:5484] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 3f156b7cfb4f40f481699b257153e5cb reported cstate change: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.407668 6915 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b10e180c11c4cd4a308fc7d311840c2 with delete_type TABLET_DATA_TOMBSTONED (TS 137e2183ff57411791e9af58dfab0a3c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58498
I20240502 01:54:29.408308 10115 tablet_replica.cc:330] T 4b10e180c11c4cd4a308fc7d311840c2 P 137e2183ff57411791e9af58dfab0a3c: stopping tablet replica
I20240502 01:54:29.408535 10115 raft_consensus.cc:2229] T 4b10e180c11c4cd4a308fc7d311840c2 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:29.408833 10115 raft_consensus.cc:2258] T 4b10e180c11c4cd4a308fc7d311840c2 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:29.410043 10115 ts_tablet_manager.cc:1875] T 4b10e180c11c4cd4a308fc7d311840c2 P 137e2183ff57411791e9af58dfab0a3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:29.415235 6782 tablet_service.cc:1514] Processing DeleteTablet for tablet 46b6cd134f7e4b2fb56f4df44ed00f15 with delete_type TABLET_DATA_TOMBSTONED (TS 8214b6c4de6d4aa58a22af551eb8dedd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35388
I20240502 01:54:29.415655 10070 tablet_replica.cc:330] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 8214b6c4de6d4aa58a22af551eb8dedd: stopping tablet replica
I20240502 01:54:29.415685 10114 consensus_queue.cc:1035] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:29.415894 10070 raft_consensus.cc:2229] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:29.416265 10070 raft_consensus.cc:2258] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:29.417438 10070 ts_tablet_manager.cc:1875] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:29.419075 7198 raft_consensus.cc:2932] T 46b6cd134f7e4b2fb56f4df44ed00f15 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.422621 10070 ts_tablet_manager.cc:1888] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 8214b6c4de6d4aa58a22af551eb8dedd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:29.422722 10070 log.cc:1196] T 46b6cd134f7e4b2fb56f4df44ed00f15 P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/46b6cd134f7e4b2fb56f4df44ed00f15
I20240502 01:54:29.423537 10115 ts_tablet_manager.cc:1888] T 4b10e180c11c4cd4a308fc7d311840c2 P 137e2183ff57411791e9af58dfab0a3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:29.423624 6645 catalog_manager.cc:4830] TS 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): tablet 46b6cd134f7e4b2fb56f4df44ed00f15 (table rebalance_test_table_3 [id=76b4b12b25de439c840462959d467432]) successfully deleted
I20240502 01:54:29.423815 10115 log.cc:1196] T 4b10e180c11c4cd4a308fc7d311840c2 P 137e2183ff57411791e9af58dfab0a3c: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/wals/4b10e180c11c4cd4a308fc7d311840c2
I20240502 01:54:29.424595 6646 catalog_manager.cc:4830] TS 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315): tablet 4b10e180c11c4cd4a308fc7d311840c2 (table rebalance_test_table_8 [id=8abdf44bc9e2414b9e4f80bf4c87c953]) successfully deleted
I20240502 01:54:29.429757 6935 consensus_queue.cc:237] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:29.432202 7067 raft_consensus.cc:1261] T 98630e8d517a4089b236f32f7616cb00 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:29.432214 6803 raft_consensus.cc:1261] T 98630e8d517a4089b236f32f7616cb00 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:29.432917 10074 consensus_queue.cc:1035] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:29.436678 9878 raft_consensus.cc:2932] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:29.437727 9395 consensus_queue.cc:1035] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:29.439635 7067 raft_consensus.cc:2932] T 98630e8d517a4089b236f32f7616cb00 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:29.440708 6803 raft_consensus.cc:2932] T 98630e8d517a4089b236f32f7616cb00 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:29.442813 8957 raft_consensus.cc:1203] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:29.447135 6654 catalog_manager.cc:5484] T 98630e8d517a4089b236f32f7616cb00 P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
W20240502 01:54:29.449131 6871 consensus_peers.cc:485] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c -> Peer c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199:35245): Couldn't send request to peer c39b0639d5fb4d9da09bf1d67d617fcc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98630e8d517a4089b236f32f7616cb00. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:29.458475 9274 raft_consensus.cc:1050] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17: attempting to promote NON_VOTER 3f156b7cfb4f40f481699b257153e5cb to VOTER
I20240502 01:54:29.459302 9274 consensus_queue.cc:237] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:29.461838 6803 raft_consensus.cc:1261] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.461897 8692 raft_consensus.cc:1261] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.461902 7067 raft_consensus.cc:1261] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.462503 10245 consensus_queue.cc:1035] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.462913 10059 consensus_queue.cc:1035] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.463227 9274 consensus_queue.cc:1035] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.464090 6935 consensus_queue.cc:237] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:29.466842 7066 raft_consensus.cc:1261] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:29.467350 9878 consensus_queue.cc:1035] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:29.467736 6872 consensus_peers.cc:485] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c -> Peer 48866680637c40368e45ba844efb9298 (127.31.161.200:34863): Couldn't send request to peer 48866680637c40368e45ba844efb9298. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76fb5e1f69284140a6c6da264a124b3e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:29.467804 7198 raft_consensus.cc:1261] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:29.468909 9878 consensus_queue.cc:1035] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:29.471113 9395 raft_consensus.cc:2932] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:29.471920 7066 raft_consensus.cc:2932] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:29.474105 7198 raft_consensus.cc:2932] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:29.478159 6653 catalog_manager.cc:5484] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c reported cstate change: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:29.494689 10245 raft_consensus.cc:2932] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.498927 9087 raft_consensus.cc:1203] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:29.499683 6657 catalog_manager.cc:5484] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:29.501098 6803 raft_consensus.cc:2932] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.501293 7067 raft_consensus.cc:2932] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.504001 7198 consensus_queue.cc:237] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:29.506664 8691 raft_consensus.cc:1261] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.507360 10245 consensus_queue.cc:1035] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.508423 8692 raft_consensus.cc:2932] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.512220 7066 raft_consensus.cc:1261] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.513293 10245 consensus_queue.cc:1035] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:29.515336 9138 raft_consensus.cc:2932] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.516305 7066 raft_consensus.cc:2932] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.520598 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d55da66aad2140d788704b056e476e99 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:29.520520 8692 raft_consensus.cc:2932] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.522043 8691 raft_consensus.cc:1203] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:29.522843 6653 catalog_manager.cc:5484] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.527031 6782 tablet_service.cc:1514] Processing DeleteTablet for tablet d55da66aad2140d788704b056e476e99 with delete_type TABLET_DATA_TOMBSTONED (TS 8214b6c4de6d4aa58a22af551eb8dedd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35388
I20240502 01:54:29.528007 10070 tablet_replica.cc:330] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd: stopping tablet replica
I20240502 01:54:29.528251 10070 raft_consensus.cc:2229] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:29.528576 10070 raft_consensus.cc:2258] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:29.530251 10070 ts_tablet_manager.cc:1875] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:29.535249 10070 ts_tablet_manager.cc:1888] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:29.535410 10070 log.cc:1196] T d55da66aad2140d788704b056e476e99 P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/d55da66aad2140d788704b056e476e99
I20240502 01:54:29.536167 6645 catalog_manager.cc:4830] TS 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): tablet d55da66aad2140d788704b056e476e99 (table rebalance_test_table_0 [id=145808b6c1a445a19f5a84827cd55658]) successfully deleted
I20240502 01:54:29.536723 7198 consensus_queue.cc:237] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:29.539062 7066 raft_consensus.cc:1261] T ee804f1981f84a118051996fb83056b8 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:29.539137 7134 consensus_peers.cc:485] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 -> Peer 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198:37123): Couldn't send request to peer 9e47e5c098b848c9abe1559cbcbb6321. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee804f1981f84a118051996fb83056b8. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:29.539132 6803 raft_consensus.cc:1261] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:29.539656 9274 consensus_queue.cc:1035] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:29.539953 9138 consensus_queue.cc:1035] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:29.542528 9274 raft_consensus.cc:2932] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:29.543131 6803 raft_consensus.cc:2932] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:29.543797 7066 raft_consensus.cc:2932] T ee804f1981f84a118051996fb83056b8 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:29.545346 10054 heartbeater.cc:503] Master 127.31.161.254:36997 was elected leader, sending a full tablet report...
I20240502 01:54:29.547879 10245 raft_consensus.cc:1050] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17: attempting to promote NON_VOTER 48866680637c40368e45ba844efb9298 to VOTER
I20240502 01:54:29.548305 6654 catalog_manager.cc:5484] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index -1 to 4, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:29.548718 10245 consensus_queue.cc:237] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:29.551548 9087 raft_consensus.cc:1261] T ae02d485bc56444e994a56f759fbed1f P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.551606 6935 raft_consensus.cc:1261] T ae02d485bc56444e994a56f759fbed1f P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.552222 7067 raft_consensus.cc:1261] T ae02d485bc56444e994a56f759fbed1f P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.552311 10059 consensus_queue.cc:1035] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.552824 10059 consensus_queue.cc:1035] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.553112 9138 consensus_queue.cc:1035] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.567729 9274 raft_consensus.cc:2932] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.568563 6935 raft_consensus.cc:2932] T ae02d485bc56444e994a56f759fbed1f P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.569869 7066 raft_consensus.cc:2932] T ae02d485bc56444e994a56f759fbed1f P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.573633 6653 catalog_manager.cc:5484] T ae02d485bc56444e994a56f759fbed1f P 137e2183ff57411791e9af58dfab0a3c reported cstate change: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.575165 9087 raft_consensus.cc:2932] T ae02d485bc56444e994a56f759fbed1f P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.580929 7198 consensus_queue.cc:237] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:29.583992 9087 raft_consensus.cc:1261] T ae02d485bc56444e994a56f759fbed1f P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.584640 10245 consensus_queue.cc:1035] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:29.584846 6935 raft_consensus.cc:1261] T ae02d485bc56444e994a56f759fbed1f P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.585525 10245 consensus_queue.cc:1035] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:29.587335 10261 raft_consensus.cc:2932] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.587970 9087 raft_consensus.cc:2932] T ae02d485bc56444e994a56f759fbed1f P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.588997 6935 raft_consensus.cc:2932] T ae02d485bc56444e994a56f759fbed1f P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.590953 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ae02d485bc56444e994a56f759fbed1f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:29.592501 6657 catalog_manager.cc:5484] T ae02d485bc56444e994a56f759fbed1f P 48866680637c40368e45ba844efb9298 reported cstate change: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.595256 7066 consensus_queue.cc:237] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:29.596468 7047 tablet_service.cc:1514] Processing DeleteTablet for tablet ae02d485bc56444e994a56f759fbed1f with delete_type TABLET_DATA_TOMBSTONED (TS 11c097d7d32a4c22a6385a7465876eb3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59762
I20240502 01:54:29.599339 6803 raft_consensus.cc:1261] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:29.600065 10114 consensus_queue.cc:1035] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:29.600364 7003 consensus_peers.cc:485] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 -> Peer c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199:35245): Couldn't send request to peer c39b0639d5fb4d9da09bf1d67d617fcc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43e6e5b3435a446499106efe7baf92aa. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:29.601490 10271 tablet_replica.cc:330] T ae02d485bc56444e994a56f759fbed1f P 11c097d7d32a4c22a6385a7465876eb3: stopping tablet replica
I20240502 01:54:29.601516 6935 raft_consensus.cc:1261] T 43e6e5b3435a446499106efe7baf92aa P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:29.601709 10271 raft_consensus.cc:2229] T ae02d485bc56444e994a56f759fbed1f P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:29.602129 10271 raft_consensus.cc:2258] T ae02d485bc56444e994a56f759fbed1f P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:29.602051 10114 consensus_queue.cc:1035] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:29.602962 10112 raft_consensus.cc:2932] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:29.603410 10271 ts_tablet_manager.cc:1875] T ae02d485bc56444e994a56f759fbed1f P 11c097d7d32a4c22a6385a7465876eb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:29.603624 6803 raft_consensus.cc:2932] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:29.609287 6653 catalog_manager.cc:5484] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:29.609685 6935 raft_consensus.cc:2932] T 43e6e5b3435a446499106efe7baf92aa P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:29.612308 10271 ts_tablet_manager.cc:1888] T ae02d485bc56444e994a56f759fbed1f P 11c097d7d32a4c22a6385a7465876eb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:29.612469 10271 log.cc:1196] T ae02d485bc56444e994a56f759fbed1f P 11c097d7d32a4c22a6385a7465876eb3: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/wals/ae02d485bc56444e994a56f759fbed1f
I20240502 01:54:29.614152 6646 catalog_manager.cc:4830] TS 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083): tablet ae02d485bc56444e994a56f759fbed1f (table rebalance_test_table_7 [id=4d5b7e86d6b24299a377cfaff5756f47]) successfully deleted
I20240502 01:54:29.616384 10179 raft_consensus.cc:1050] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd: attempting to promote NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc to VOTER
I20240502 01:54:29.617303 10179 consensus_queue.cc:237] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:29.620254 6935 raft_consensus.cc:1261] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.620507 8957 raft_consensus.cc:1261] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.620862 9516 consensus_queue.cc:1035] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.621202 10179 consensus_queue.cc:1035] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.621802 7067 raft_consensus.cc:1261] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.623848 9516 consensus_queue.cc:1035] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.627611 9516 raft_consensus.cc:2932] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:29.630656 7067 raft_consensus.cc:2932] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:29.634068 6935 consensus_queue.cc:237] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:29.634235 10179 raft_consensus.cc:1050] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd: attempting to promote NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 to VOTER
I20240502 01:54:29.635020 10179 consensus_queue.cc:237] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:29.638065 6657 catalog_manager.cc:5484] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:29.643092 8824 raft_consensus.cc:1203] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:29.648000 8824 raft_consensus.cc:1261] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.649606 6936 raft_consensus.cc:1261] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.651095 7199 raft_consensus.cc:1261] T a14c3c945b724df5ac3f917cef3186e0 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.651885 10286 consensus_queue.cc:1035] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.653416 10286 consensus_queue.cc:1035] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.656216 10283 consensus_queue.cc:1035] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.657456 8957 raft_consensus.cc:2932] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:29.663547 7198 raft_consensus.cc:1261] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:29.664188 6803 raft_consensus.cc:1261] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:29.664247 9395 consensus_queue.cc:1035] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:29.666100 9395 consensus_queue.cc:1035] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:29.667019 6872 consensus_peers.cc:485] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c -> Peer 48866680637c40368e45ba844efb9298 (127.31.161.200:34863): Couldn't send request to peer 48866680637c40368e45ba844efb9298. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8960ca8e5d26429589abc4a154311983. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:29.668605 6801 consensus_queue.cc:237] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:29.672013 6935 raft_consensus.cc:2932] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:29.674031 8957 raft_consensus.cc:1261] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.674989 9516 consensus_queue.cc:1035] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:29.678689 10287 raft_consensus.cc:2932] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:29.679343 8956 raft_consensus.cc:2932] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:29.682986 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d1f4c94a105741339ba69e641244ac18 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:29.685019 6657 catalog_manager.cc:5484] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc reported cstate change: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:29.687206 6937 raft_consensus.cc:1261] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.687953 10287 consensus_queue.cc:1035] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:29.690670 7047 tablet_service.cc:1514] Processing DeleteTablet for tablet d1f4c94a105741339ba69e641244ac18 with delete_type TABLET_DATA_TOMBSTONED (TS 11c097d7d32a4c22a6385a7465876eb3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59762
I20240502 01:54:29.691107 10271 tablet_replica.cc:330] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3: stopping tablet replica
I20240502 01:54:29.691327 10271 raft_consensus.cc:2229] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:29.691687 10271 raft_consensus.cc:2258] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:29.692719 10271 ts_tablet_manager.cc:1875] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:29.698935 10287 raft_consensus.cc:2932] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.701718 7197 raft_consensus.cc:2932] T a14c3c945b724df5ac3f917cef3186e0 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.706974 10283 raft_consensus.cc:1050] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd: attempting to promote NON_VOTER 48866680637c40368e45ba844efb9298 to VOTER
I20240502 01:54:29.707850 10283 consensus_queue.cc:237] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:29.712191 6937 raft_consensus.cc:2932] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.713864 8824 raft_consensus.cc:2932] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.714474 6653 catalog_manager.cc:5484] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:29.718072 10271 ts_tablet_manager.cc:1888] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:29.718227 10271 log.cc:1196] T d1f4c94a105741339ba69e641244ac18 P 11c097d7d32a4c22a6385a7465876eb3: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/wals/d1f4c94a105741339ba69e641244ac18
I20240502 01:54:29.719180 6646 catalog_manager.cc:4830] TS 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083): tablet d1f4c94a105741339ba69e641244ac18 (table rebalance_test_table_5 [id=abae7d7cdd254582beebaba1abe05ac1]) successfully deleted
I20240502 01:54:29.724522 6801 consensus_queue.cc:237] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:29.726596 8824 raft_consensus.cc:1261] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.727862 10179 consensus_queue.cc:1035] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:29.728046 6937 raft_consensus.cc:1261] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.729527 6935 raft_consensus.cc:2932] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:29.728622 10287 consensus_queue.cc:1035] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20240502 01:54:29.731391 9395 raft_consensus.cc:2932] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:29.732436 10287 raft_consensus.cc:2932] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.733549 7198 raft_consensus.cc:2932] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:29.737252 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a14c3c945b724df5ac3f917cef3186e0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:29.737910 6937 raft_consensus.cc:2932] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.739631 6657 catalog_manager.cc:5484] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index 5 to 6, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:29.741940 8824 raft_consensus.cc:2932] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:29.745307 7179 tablet_service.cc:1514] Processing DeleteTablet for tablet a14c3c945b724df5ac3f917cef3186e0 with delete_type TABLET_DATA_TOMBSTONED (TS fc21155d36d5416fa3650f0268a20b17 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40840
I20240502 01:54:29.745749 10193 tablet_replica.cc:330] T a14c3c945b724df5ac3f917cef3186e0 P fc21155d36d5416fa3650f0268a20b17: stopping tablet replica
I20240502 01:54:29.745947 10193 raft_consensus.cc:2229] T a14c3c945b724df5ac3f917cef3186e0 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:29.746168 10193 raft_consensus.cc:2258] T a14c3c945b724df5ac3f917cef3186e0 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:29.749379 6657 catalog_manager.cc:5484] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:29.750964 7197 raft_consensus.cc:1261] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.751536 10283 consensus_queue.cc:1035] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 01:54:29.752374 9086 raft_consensus.cc:1261] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.757370 8246 consensus_queue.cc:1035] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.758661 10193 ts_tablet_manager.cc:1875] T a14c3c945b724df5ac3f917cef3186e0 P fc21155d36d5416fa3650f0268a20b17: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:29.758385 6803 raft_consensus.cc:2932] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:29.761933 6935 raft_consensus.cc:1261] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.762591 8246 consensus_queue.cc:1035] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.768923 10193 ts_tablet_manager.cc:1888] T a14c3c945b724df5ac3f917cef3186e0 P fc21155d36d5416fa3650f0268a20b17: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:29.769042 10193 log.cc:1196] T a14c3c945b724df5ac3f917cef3186e0 P fc21155d36d5416fa3650f0268a20b17: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/a14c3c945b724df5ac3f917cef3186e0
I20240502 01:54:29.769783 6645 catalog_manager.cc:4830] TS fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873): tablet a14c3c945b724df5ac3f917cef3186e0 (table rebalance_test_table_4 [id=9231a7576c804559bc4a4c54496d5223]) successfully deleted
I20240502 01:54:29.777630 10245 raft_consensus.cc:1050] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17: attempting to promote NON_VOTER 3f156b7cfb4f40f481699b257153e5cb to VOTER
I20240502 01:54:29.778358 10245 consensus_queue.cc:237] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:29.783160 10179 raft_consensus.cc:2932] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.784703 6935 raft_consensus.cc:2932] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.785408 8692 raft_consensus.cc:1261] T 482f77b65a254ed3aec717c06698ffa7 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.785435 7067 raft_consensus.cc:1261] T 482f77b65a254ed3aec717c06698ffa7 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.786017 9274 consensus_queue.cc:1035] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.786322 6937 raft_consensus.cc:1261] T 482f77b65a254ed3aec717c06698ffa7 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.786341 9087 raft_consensus.cc:2932] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.787885 9274 consensus_queue.cc:1035] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.790370 10245 consensus_queue.cc:1035] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.792695 6654 catalog_manager.cc:5484] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c reported cstate change: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.793546 7197 raft_consensus.cc:2932] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.799131 6803 consensus_queue.cc:237] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:29.802042 9274 raft_consensus.cc:2932] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.803458 6935 raft_consensus.cc:1261] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.804035 8246 consensus_queue.cc:1035] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:29.805526 9086 raft_consensus.cc:1261] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.805351 6935 raft_consensus.cc:2932] T 482f77b65a254ed3aec717c06698ffa7 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.806648 10283 consensus_queue.cc:1035] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:29.807343 6936 raft_consensus.cc:2932] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.808123 9086 raft_consensus.cc:2932] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.806375 10179 raft_consensus.cc:2932] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.812703 6657 catalog_manager.cc:5484] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 reported cstate change: config changed from index 5 to 6, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.814184 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 39f0a7df4f624eee81c25b41d19d1740 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:29.815568 8692 raft_consensus.cc:2932] T 482f77b65a254ed3aec717c06698ffa7 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.816905 6653 catalog_manager.cc:5484] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:29.818531 7179 tablet_service.cc:1514] Processing DeleteTablet for tablet 39f0a7df4f624eee81c25b41d19d1740 with delete_type TABLET_DATA_TOMBSTONED (TS fc21155d36d5416fa3650f0268a20b17 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40840
I20240502 01:54:29.818876 10193 tablet_replica.cc:330] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17: stopping tablet replica
I20240502 01:54:29.818681 7067 raft_consensus.cc:2932] T 482f77b65a254ed3aec717c06698ffa7 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.819059 10193 raft_consensus.cc:2229] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:29.819331 10193 raft_consensus.cc:2258] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:29.820245 7197 consensus_queue.cc:237] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:29.820636 10193 ts_tablet_manager.cc:1875] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:29.821817 7198 consensus_queue.cc:237] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:29.824254 7066 raft_consensus.cc:1261] T 482f77b65a254ed3aec717c06698ffa7 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.824795 9274 consensus_queue.cc:1035] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:29.826144 8691 raft_consensus.cc:1261] T 482f77b65a254ed3aec717c06698ffa7 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.828862 9274 consensus_queue.cc:1035] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:29.832224 9274 raft_consensus.cc:2932] T 482f77b65a254ed3aec717c06698ffa7 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.833245 8692 raft_consensus.cc:2932] T 482f77b65a254ed3aec717c06698ffa7 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.832702 10193 ts_tablet_manager.cc:1888] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:29.836668 10193 log.cc:1196] T 39f0a7df4f624eee81c25b41d19d1740 P fc21155d36d5416fa3650f0268a20b17: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/39f0a7df4f624eee81c25b41d19d1740
I20240502 01:54:29.837297 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 482f77b65a254ed3aec717c06698ffa7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:29.838117 6645 catalog_manager.cc:4830] TS fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873): tablet 39f0a7df4f624eee81c25b41d19d1740 (table rebalance_test_table_0 [id=145808b6c1a445a19f5a84827cd55658]) successfully deleted
I20240502 01:54:29.838795 6657 catalog_manager.cc:5484] T 482f77b65a254ed3aec717c06698ffa7 P 3f156b7cfb4f40f481699b257153e5cb reported cstate change: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.842715 6915 tablet_service.cc:1514] Processing DeleteTablet for tablet 482f77b65a254ed3aec717c06698ffa7 with delete_type TABLET_DATA_TOMBSTONED (TS 137e2183ff57411791e9af58dfab0a3c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58498
I20240502 01:54:29.842509 7066 raft_consensus.cc:2932] T 482f77b65a254ed3aec717c06698ffa7 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:29.842926 6936 raft_consensus.cc:1261] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:29.843058 10115 tablet_replica.cc:330] T 482f77b65a254ed3aec717c06698ffa7 P 137e2183ff57411791e9af58dfab0a3c: stopping tablet replica
I20240502 01:54:29.843241 10115 raft_consensus.cc:2229] T 482f77b65a254ed3aec717c06698ffa7 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:29.843521 10115 raft_consensus.cc:2258] T 482f77b65a254ed3aec717c06698ffa7 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:29.844273 10115 ts_tablet_manager.cc:1875] T 482f77b65a254ed3aec717c06698ffa7 P 137e2183ff57411791e9af58dfab0a3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:29.849709 10245 consensus_queue.cc:1035] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:29.850052 7134 consensus_peers.cc:485] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 -> Peer 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197:44451): Couldn't send request to peer 3f156b7cfb4f40f481699b257153e5cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70ce4bbee414402aa97c9f3914b9d0c6. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:29.850879 10115 ts_tablet_manager.cc:1888] T 482f77b65a254ed3aec717c06698ffa7 P 137e2183ff57411791e9af58dfab0a3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:29.851024 10115 log.cc:1196] T 482f77b65a254ed3aec717c06698ffa7 P 137e2183ff57411791e9af58dfab0a3c: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/wals/482f77b65a254ed3aec717c06698ffa7
I20240502 01:54:29.851879 7067 raft_consensus.cc:1261] T 70ce4bbee414402aa97c9f3914b9d0c6 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:29.851868 6646 catalog_manager.cc:4830] TS 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315): tablet 482f77b65a254ed3aec717c06698ffa7 (table rebalance_test_table_5 [id=abae7d7cdd254582beebaba1abe05ac1]) successfully deleted
I20240502 01:54:29.852401 10245 consensus_queue.cc:1035] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:29.852890 9274 raft_consensus.cc:2932] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:29.853665 6936 raft_consensus.cc:2932] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:29.858647 6654 catalog_manager.cc:5484] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:29.858644 7067 raft_consensus.cc:2932] T 70ce4bbee414402aa97c9f3914b9d0c6 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:29.934669 9878 raft_consensus.cc:1050] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c: attempting to promote NON_VOTER 48866680637c40368e45ba844efb9298 to VOTER
I20240502 01:54:29.935647 9878 consensus_queue.cc:237] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:29.938486 9086 raft_consensus.cc:1261] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.940183 7197 raft_consensus.cc:1261] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.940204 6803 raft_consensus.cc:1261] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.940800 9395 consensus_queue.cc:1035] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.942966 10074 consensus_queue.cc:1035] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.943473 10074 consensus_queue.cc:1035] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.958134 9395 raft_consensus.cc:2932] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.960100 9086 raft_consensus.cc:2932] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.963577 10349 ts_tablet_manager.cc:902] T ee804f1981f84a118051996fb83056b8 P 9e47e5c098b848c9abe1559cbcbb6321: Initiating tablet copy from peer fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873)
I20240502 01:54:29.963191 7197 raft_consensus.cc:2932] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.965766 6659 catalog_manager.cc:5484] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298 reported cstate change: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.967104 8246 raft_consensus.cc:1050] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd: attempting to promote NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc to VOTER
I20240502 01:54:29.967927 8246 consensus_queue.cc:237] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:29.971617 6801 raft_consensus.cc:2932] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.973335 7067 raft_consensus.cc:1261] T c303b823d03249f89bb6dfaca3934cc9 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.973353 8956 raft_consensus.cc:1261] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.973704 6936 consensus_queue.cc:237] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:29.974035 10286 consensus_queue.cc:1035] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.974620 10286 consensus_queue.cc:1035] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.976403 6936 raft_consensus.cc:1261] T c303b823d03249f89bb6dfaca3934cc9 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:29.976948 7197 raft_consensus.cc:1261] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.977895 10288 consensus_queue.cc:1035] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:29.978101 9395 consensus_queue.cc:1035] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:29.981173 9878 raft_consensus.cc:2932] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.982992 9086 raft_consensus.cc:1261] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:29.985231 10074 consensus_queue.cc:1035] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:29.987320 7197 raft_consensus.cc:2932] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:29.990796 8246 raft_consensus.cc:2932] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:29.992105 6935 raft_consensus.cc:2932] T c303b823d03249f89bb6dfaca3934cc9 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:29.992660 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dea582c2912241609405695eed436abe with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:29.996727 7067 raft_consensus.cc:2932] T c303b823d03249f89bb6dfaca3934cc9 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:29.996909 6657 catalog_manager.cc:5484] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c reported cstate change: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:29.999912 9086 raft_consensus.cc:2932] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:30.000308 8956 raft_consensus.cc:2932] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:30.004608 6782 tablet_service.cc:1514] Processing DeleteTablet for tablet dea582c2912241609405695eed436abe with delete_type TABLET_DATA_TOMBSTONED (TS 8214b6c4de6d4aa58a22af551eb8dedd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35388
I20240502 01:54:30.005144 10070 tablet_replica.cc:330] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd: stopping tablet replica
I20240502 01:54:30.005376 10070 raft_consensus.cc:2229] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:30.005617 10070 raft_consensus.cc:2258] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:30.004051 6657 catalog_manager.cc:5484] T c303b823d03249f89bb6dfaca3934cc9 P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:30.013336 10070 ts_tablet_manager.cc:1875] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:30.016269 6801 consensus_queue.cc:237] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:30.017694 10349 tablet_copy_client.cc:308] T ee804f1981f84a118051996fb83056b8 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:38873
I20240502 01:54:30.018839 8956 raft_consensus.cc:1261] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:30.019204 7218 tablet_copy_service.cc:140] P fc21155d36d5416fa3650f0268a20b17: Received BeginTabletCopySession request for tablet ee804f1981f84a118051996fb83056b8 from peer 9e47e5c098b848c9abe1559cbcbb6321 ({username='slave'} at 127.31.161.198:52833)
I20240502 01:54:30.019348 7218 tablet_copy_service.cc:161] P fc21155d36d5416fa3650f0268a20b17: Beginning new tablet copy session on tablet ee804f1981f84a118051996fb83056b8 from peer 9e47e5c098b848c9abe1559cbcbb6321 at {username='slave'} at 127.31.161.198:52833: session id = 9e47e5c098b848c9abe1559cbcbb6321-ee804f1981f84a118051996fb83056b8
I20240502 01:54:30.022392 7218 tablet_copy_source_session.cc:205] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:30.023146 7066 raft_consensus.cc:1261] T c303b823d03249f89bb6dfaca3934cc9 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:30.024078 10288 consensus_queue.cc:1035] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:30.024714 10288 consensus_queue.cc:1035] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:30.024977 10349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee804f1981f84a118051996fb83056b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:30.025224 10363 ts_tablet_manager.cc:902] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298: Initiating tablet copy from peer 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:30.028396 10288 raft_consensus.cc:2932] T c303b823d03249f89bb6dfaca3934cc9 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:30.031572 10349 tablet_copy_client.cc:783] T ee804f1981f84a118051996fb83056b8 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:30.031811 10349 tablet_copy_client.cc:647] T ee804f1981f84a118051996fb83056b8 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:30.032819 8956 raft_consensus.cc:2932] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:30.037257 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c303b823d03249f89bb6dfaca3934cc9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:30.038440 6657 catalog_manager.cc:5484] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc reported cstate change: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:30.039148 7066 raft_consensus.cc:2932] T c303b823d03249f89bb6dfaca3934cc9 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:30.040818 10361 tablet_copy_client.cc:1002] T ee804f1981f84a118051996fb83056b8 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:30.043087 6915 tablet_service.cc:1514] Processing DeleteTablet for tablet c303b823d03249f89bb6dfaca3934cc9 with delete_type TABLET_DATA_TOMBSTONED (TS 137e2183ff57411791e9af58dfab0a3c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58498
I20240502 01:54:30.043452 10115 tablet_replica.cc:330] T c303b823d03249f89bb6dfaca3934cc9 P 137e2183ff57411791e9af58dfab0a3c: stopping tablet replica
I20240502 01:54:30.043677 10115 raft_consensus.cc:2229] T c303b823d03249f89bb6dfaca3934cc9 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:30.043977 10115 raft_consensus.cc:2258] T c303b823d03249f89bb6dfaca3934cc9 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:30.044116 10070 ts_tablet_manager.cc:1888] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:30.045058 10115 ts_tablet_manager.cc:1875] T c303b823d03249f89bb6dfaca3934cc9 P 137e2183ff57411791e9af58dfab0a3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:30.050328 10070 log.cc:1196] T dea582c2912241609405695eed436abe P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/dea582c2912241609405695eed436abe
I20240502 01:54:30.051159 6645 catalog_manager.cc:4830] TS 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): tablet dea582c2912241609405695eed436abe (table rebalance_test_table_2 [id=3316a56a8d6043c6b05160d6e55a33e0]) successfully deleted
I20240502 01:54:30.052909 10115 ts_tablet_manager.cc:1888] T c303b823d03249f89bb6dfaca3934cc9 P 137e2183ff57411791e9af58dfab0a3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:30.054244 10115 log.cc:1196] T c303b823d03249f89bb6dfaca3934cc9 P 137e2183ff57411791e9af58dfab0a3c: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/wals/c303b823d03249f89bb6dfaca3934cc9
I20240502 01:54:30.055034 6646 catalog_manager.cc:4830] TS 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315): tablet c303b823d03249f89bb6dfaca3934cc9 (table rebalance_test_table_1 [id=9a08c31578ae4224915289c7c289ee7a]) successfully deleted
I20240502 01:54:30.060266 10368 ts_tablet_manager.cc:902] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc: Initiating tablet copy from peer 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:30.061096 10363 tablet_copy_client.cc:308] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.194:45315
I20240502 01:54:30.061810 6958 tablet_copy_service.cc:140] P 137e2183ff57411791e9af58dfab0a3c: Received BeginTabletCopySession request for tablet 76fb5e1f69284140a6c6da264a124b3e from peer 48866680637c40368e45ba844efb9298 ({username='slave'} at 127.31.161.200:55333)
I20240502 01:54:30.061986 6958 tablet_copy_service.cc:161] P 137e2183ff57411791e9af58dfab0a3c: Beginning new tablet copy session on tablet 76fb5e1f69284140a6c6da264a124b3e from peer 48866680637c40368e45ba844efb9298 at {username='slave'} at 127.31.161.200:55333: session id = 48866680637c40368e45ba844efb9298-76fb5e1f69284140a6c6da264a124b3e
I20240502 01:54:30.064119 6958 tablet_copy_source_session.cc:205] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:30.065798 10363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76fb5e1f69284140a6c6da264a124b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:30.070547 10363 tablet_copy_client.cc:783] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:30.070768 10363 tablet_copy_client.cc:647] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:30.071339 10368 tablet_copy_client.cc:308] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.194:45315
I20240502 01:54:30.072551 10374 tablet_copy_client.cc:1002] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:30.078688 6958 tablet_copy_service.cc:140] P 137e2183ff57411791e9af58dfab0a3c: Received BeginTabletCopySession request for tablet 98630e8d517a4089b236f32f7616cb00 from peer c39b0639d5fb4d9da09bf1d67d617fcc ({username='slave'} at 127.31.161.199:41907)
I20240502 01:54:30.078845 6958 tablet_copy_service.cc:161] P 137e2183ff57411791e9af58dfab0a3c: Beginning new tablet copy session on tablet 98630e8d517a4089b236f32f7616cb00 from peer c39b0639d5fb4d9da09bf1d67d617fcc at {username='slave'} at 127.31.161.199:41907: session id = c39b0639d5fb4d9da09bf1d67d617fcc-98630e8d517a4089b236f32f7616cb00
I20240502 01:54:30.081182 6958 tablet_copy_source_session.cc:205] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:30.082414 10368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98630e8d517a4089b236f32f7616cb00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:30.088928 10368 tablet_copy_client.cc:783] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:30.089213 10368 tablet_copy_client.cc:647] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:30.090600 10378 tablet_copy_client.cc:1002] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:30.092661 10283 raft_consensus.cc:1050] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd: attempting to promote NON_VOTER 3f156b7cfb4f40f481699b257153e5cb to VOTER
I20240502 01:54:30.093665 10283 consensus_queue.cc:237] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:30.095861 7197 raft_consensus.cc:1261] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:30.096017 8692 raft_consensus.cc:1261] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:30.096258 7067 raft_consensus.cc:1261] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:30.096850 10288 consensus_queue.cc:1035] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:30.097187 10283 consensus_queue.cc:1035] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:30.097429 8246 consensus_queue.cc:1035] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:30.100757 8246 raft_consensus.cc:2932] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:30.105504 8692 raft_consensus.cc:2932] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:30.107230 6654 catalog_manager.cc:5484] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:30.108475 7197 raft_consensus.cc:2932] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:30.112582 6801 consensus_queue.cc:237] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:30.118593 8691 raft_consensus.cc:1261] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:30.119349 7197 raft_consensus.cc:1261] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:30.119853 8246 consensus_queue.cc:1035] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:30.121763 8246 consensus_queue.cc:1035] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:30.122628 9138 raft_consensus.cc:1050] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17: attempting to promote NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 to VOTER
I20240502 01:54:30.122997 10288 raft_consensus.cc:2932] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:30.123430 9138 consensus_queue.cc:237] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:30.126205 8824 raft_consensus.cc:1261] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:30.126267 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 945224f5cff74edb9447bebc743e5086 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:30.126302 7067 raft_consensus.cc:1261] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:30.126852 9274 consensus_queue.cc:1035] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:30.126737 7197 raft_consensus.cc:2932] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:30.128033 9274 consensus_queue.cc:1035] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:30.128949 6801 raft_consensus.cc:1261] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:30.130599 9274 consensus_queue.cc:1035] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:30.132225 8691 raft_consensus.cc:2932] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:30.132597 6657 catalog_manager.cc:5484] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:30.138193 9274 raft_consensus.cc:2932] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:30.138603 7047 tablet_service.cc:1514] Processing DeleteTablet for tablet 945224f5cff74edb9447bebc743e5086 with delete_type TABLET_DATA_TOMBSTONED (TS 11c097d7d32a4c22a6385a7465876eb3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59762
I20240502 01:54:30.138967 10271 tablet_replica.cc:330] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3: stopping tablet replica
I20240502 01:54:30.139180 10271 raft_consensus.cc:2229] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:30.139410 10271 pending_rounds.cc:62] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3: Trying to abort 1 pending ops.
I20240502 01:54:30.139540 10271 pending_rounds.cc:68] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023062507901038592 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "945224f5cff74edb9447bebc743e5086" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } } }
I20240502 01:54:30.139997 10271 raft_consensus.cc:2894] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 01:54:30.139477 6801 raft_consensus.cc:2932] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:30.140141 10271 raft_consensus.cc:2866] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 01:54:30.140271 10271 raft_consensus.cc:2258] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:30.141113 10271 ts_tablet_manager.cc:1875] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:30.142177 7067 raft_consensus.cc:2932] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:30.143468 8824 raft_consensus.cc:2932] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:30.148710 6657 catalog_manager.cc:5484] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:30.149446 10271 ts_tablet_manager.cc:1888] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:30.149631 10271 log.cc:1196] T 945224f5cff74edb9447bebc743e5086 P 11c097d7d32a4c22a6385a7465876eb3: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/wals/945224f5cff74edb9447bebc743e5086
I20240502 01:54:30.152179 6646 catalog_manager.cc:4830] TS 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083): tablet 945224f5cff74edb9447bebc743e5086 (table rebalance_test_table_9 [id=5b2a4358eba544959466c46759744194]) successfully deleted
I20240502 01:54:30.193434 10423 ts_tablet_manager.cc:902] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298: Initiating tablet copy from peer 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:30.196239 10423 tablet_copy_client.cc:308] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.194:45315
I20240502 01:54:30.197049 6958 tablet_copy_service.cc:140] P 137e2183ff57411791e9af58dfab0a3c: Received BeginTabletCopySession request for tablet 8960ca8e5d26429589abc4a154311983 from peer 48866680637c40368e45ba844efb9298 ({username='slave'} at 127.31.161.200:55333)
I20240502 01:54:30.197274 6958 tablet_copy_service.cc:161] P 137e2183ff57411791e9af58dfab0a3c: Beginning new tablet copy session on tablet 8960ca8e5d26429589abc4a154311983 from peer 48866680637c40368e45ba844efb9298 at {username='slave'} at 127.31.161.200:55333: session id = 48866680637c40368e45ba844efb9298-8960ca8e5d26429589abc4a154311983
I20240502 01:54:30.200254 6958 tablet_copy_source_session.cc:205] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:30.201575 10423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8960ca8e5d26429589abc4a154311983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:30.207643 10423 tablet_copy_client.cc:783] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:30.207859 10423 tablet_copy_client.cc:647] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:30.209724 10427 tablet_copy_client.cc:1002] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:30.217450 10431 ts_tablet_manager.cc:902] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc: Initiating tablet copy from peer 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:30.225492 10431 tablet_copy_client.cc:308] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:46083
I20240502 01:54:30.226215 7089 tablet_copy_service.cc:140] P 11c097d7d32a4c22a6385a7465876eb3: Received BeginTabletCopySession request for tablet 43e6e5b3435a446499106efe7baf92aa from peer c39b0639d5fb4d9da09bf1d67d617fcc ({username='slave'} at 127.31.161.199:34537)
I20240502 01:54:30.226363 7089 tablet_copy_service.cc:161] P 11c097d7d32a4c22a6385a7465876eb3: Beginning new tablet copy session on tablet 43e6e5b3435a446499106efe7baf92aa from peer c39b0639d5fb4d9da09bf1d67d617fcc at {username='slave'} at 127.31.161.199:34537: session id = c39b0639d5fb4d9da09bf1d67d617fcc-43e6e5b3435a446499106efe7baf92aa
I20240502 01:54:30.228400 7089 tablet_copy_source_session.cc:205] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:30.229630 10431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e6e5b3435a446499106efe7baf92aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:30.237246 10431 tablet_copy_client.cc:783] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:30.237510 10431 tablet_copy_client.cc:647] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:30.239135 10442 tablet_copy_client.cc:1002] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:30.261813 7067 consensus_queue.cc:237] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:30.264052 7197 raft_consensus.cc:1261] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:30.264058 6801 raft_consensus.cc:1261] T 640b6474d1834321881094ed0b11b857 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:30.264410 7004 consensus_peers.cc:485] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 -> Peer 48866680637c40368e45ba844efb9298 (127.31.161.200:34863): Couldn't send request to peer 48866680637c40368e45ba844efb9298. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 640b6474d1834321881094ed0b11b857. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:30.264632 10112 consensus_queue.cc:1035] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:30.264919 10114 consensus_queue.cc:1035] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:30.267634 10112 raft_consensus.cc:2932] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:30.268258 6801 raft_consensus.cc:2932] T 640b6474d1834321881094ed0b11b857 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:30.269655 7197 raft_consensus.cc:2932] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:30.272658 6654 catalog_manager.cc:5484] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:30.288153 6801 consensus_queue.cc:237] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:30.290468 7067 raft_consensus.cc:1261] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:30.290614 6936 raft_consensus.cc:1261] T c86cf318e8e34435bd1dac117d7cc389 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:30.290874 6738 consensus_peers.cc:485] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd -> Peer 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197:44451): Couldn't send request to peer 3f156b7cfb4f40f481699b257153e5cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c86cf318e8e34435bd1dac117d7cc389. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:30.291216 10283 consensus_queue.cc:1035] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:30.291498 8246 consensus_queue.cc:1035] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:30.294780 8246 raft_consensus.cc:2932] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:30.298736 6936 raft_consensus.cc:2932] T c86cf318e8e34435bd1dac117d7cc389 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:30.301410 7067 raft_consensus.cc:2932] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:30.302256 6654 catalog_manager.cc:5484] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:30.319983 7197 consensus_queue.cc:237] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:30.322212 7067 raft_consensus.cc:1261] T b984c722354c462882428317c411fd78 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:30.322216 6801 raft_consensus.cc:1261] T b984c722354c462882428317c411fd78 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:54:30.322705 7134 consensus_peers.cc:485] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 -> Peer 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197:44451): Couldn't send request to peer 3f156b7cfb4f40f481699b257153e5cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b984c722354c462882428317c411fd78. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:30.322779 10245 consensus_queue.cc:1035] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:30.323069 9274 consensus_queue.cc:1035] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:30.326090 10059 raft_consensus.cc:2932] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:30.329947 7067 raft_consensus.cc:2932] T b984c722354c462882428317c411fd78 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:30.335325 6658 catalog_manager.cc:5484] T b984c722354c462882428317c411fd78 P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:30.346401 6801 raft_consensus.cc:2932] T b984c722354c462882428317c411fd78 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:30.367604 6801 consensus_queue.cc:237] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:30.370107 7067 raft_consensus.cc:1261] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:30.370532 6936 raft_consensus.cc:1261] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:54:30.370637 8246 consensus_queue.cc:1035] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:54:30.371032 6739 consensus_peers.cc:485] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd -> Peer 48866680637c40368e45ba844efb9298 (127.31.161.200:34863): Couldn't send request to peer 48866680637c40368e45ba844efb9298. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c1d0974a2fd4a548806b6b6560a89f8. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:30.371189 10288 consensus_queue.cc:1035] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:54:30.374475 8246 raft_consensus.cc:2932] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:30.375078 7067 raft_consensus.cc:2932] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:30.376013 6936 raft_consensus.cc:2932] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:30.380427 6654 catalog_manager.cc:5484] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index -1 to 4, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:30.389516 10480 ts_tablet_manager.cc:902] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb: Initiating tablet copy from peer fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873)
I20240502 01:54:30.391247 10480 tablet_copy_client.cc:308] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:38873
I20240502 01:54:30.391979 7218 tablet_copy_service.cc:140] P fc21155d36d5416fa3650f0268a20b17: Received BeginTabletCopySession request for tablet 70ce4bbee414402aa97c9f3914b9d0c6 from peer 3f156b7cfb4f40f481699b257153e5cb ({username='slave'} at 127.31.161.197:48453)
I20240502 01:54:30.392168 7218 tablet_copy_service.cc:161] P fc21155d36d5416fa3650f0268a20b17: Beginning new tablet copy session on tablet 70ce4bbee414402aa97c9f3914b9d0c6 from peer 3f156b7cfb4f40f481699b257153e5cb at {username='slave'} at 127.31.161.197:48453: session id = 3f156b7cfb4f40f481699b257153e5cb-70ce4bbee414402aa97c9f3914b9d0c6
I20240502 01:54:30.394637 7218 tablet_copy_source_session.cc:205] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:30.395936 10480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ce4bbee414402aa97c9f3914b9d0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:30.402204 10480 tablet_copy_client.cc:783] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:30.402453 10480 tablet_copy_client.cc:647] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:30.403944 10489 tablet_copy_client.cc:1002] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:30.479981 7197 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d58f60c0920e4616850cb246acce1fd7"
dest_uuid: "fc21155d36d5416fa3650f0268a20b17"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39744
I20240502 01:54:30.480218 7197 raft_consensus.cc:595] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:30.558174 10245 raft_consensus.cc:979] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17: : Instructing follower 11c097d7d32a4c22a6385a7465876eb3 to start an election
I20240502 01:54:30.558399 10059 raft_consensus.cc:1067] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Signalling peer 11c097d7d32a4c22a6385a7465876eb3 to start an election
I20240502 01:54:30.559311 7067 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d58f60c0920e4616850cb246acce1fd7"
dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
from {username='slave'} at 127.31.161.196:45755
I20240502 01:54:30.559602 7067 raft_consensus.cc:482] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:30.559736 7067 raft_consensus.cc:3037] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:30.561789 7067 raft_consensus.cc:504] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:30.562443 7067 leader_election.cc:288] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 2 election: Requested vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873), 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198:37123)
I20240502 01:54:30.562851 6801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58f60c0920e4616850cb246acce1fd7" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
I20240502 01:54:30.563099 6801 raft_consensus.cc:3037] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:30.563205 7197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58f60c0920e4616850cb246acce1fd7" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fc21155d36d5416fa3650f0268a20b17"
I20240502 01:54:30.563263 8824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58f60c0920e4616850cb246acce1fd7" candidate_uuid: "11c097d7d32a4c22a6385a7465876eb3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "9e47e5c098b848c9abe1559cbcbb6321"
I20240502 01:54:30.563397 7197 raft_consensus.cc:3032] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:30.563486 7197 raft_consensus.cc:729] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Running, Role: LEADER
I20240502 01:54:30.563611 8824 raft_consensus.cc:3037] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:30.563789 7197 consensus_queue.cc:260] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:30.564271 7197 raft_consensus.cc:3037] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:30.565160 6801 raft_consensus.cc:2454] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 2.
I20240502 01:54:30.566521 8824 raft_consensus.cc:2454] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 2.
I20240502 01:54:30.567916 7002 leader_election.cc:302] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 8214b6c4de6d4aa58a22af551eb8dedd, 9e47e5c098b848c9abe1559cbcbb6321; no voters:
I20240502 01:54:30.568229 10114 raft_consensus.cc:2781] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:30.568368 7197 raft_consensus.cc:2454] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11c097d7d32a4c22a6385a7465876eb3 in term 2.
I20240502 01:54:30.568394 10114 raft_consensus.cc:686] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 2 LEADER]: Becoming Leader. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Running, Role: LEADER
I20240502 01:54:30.568661 10114 consensus_queue.cc:237] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:30.571667 6658 catalog_manager.cc:5484] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: term changed from 1 to 2, leader changed from fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) to 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195). New cstate: current_term: 2 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:30.705616 10610 ts_tablet_manager.cc:902] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298: Initiating tablet copy from peer 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:30.707091 10610 tablet_copy_client.cc:308] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:46083
I20240502 01:54:30.707891 7089 tablet_copy_service.cc:140] P 11c097d7d32a4c22a6385a7465876eb3: Received BeginTabletCopySession request for tablet 640b6474d1834321881094ed0b11b857 from peer 48866680637c40368e45ba844efb9298 ({username='slave'} at 127.31.161.200:55399)
I20240502 01:54:30.708096 7089 tablet_copy_service.cc:161] P 11c097d7d32a4c22a6385a7465876eb3: Beginning new tablet copy session on tablet 640b6474d1834321881094ed0b11b857 from peer 48866680637c40368e45ba844efb9298 at {username='slave'} at 127.31.161.200:55399: session id = 48866680637c40368e45ba844efb9298-640b6474d1834321881094ed0b11b857
I20240502 01:54:30.711171 7089 tablet_copy_source_session.cc:205] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:30.712469 10610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 640b6474d1834321881094ed0b11b857. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:30.716958 10610 tablet_copy_client.cc:783] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:30.717207 10610 tablet_copy_client.cc:647] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:30.718957 10614 tablet_copy_client.cc:1002] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:30.731052 10615 ts_tablet_manager.cc:902] T c86cf318e8e34435bd1dac117d7cc389 P 3f156b7cfb4f40f481699b257153e5cb: Initiating tablet copy from peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:30.732534 10615 tablet_copy_client.cc:308] T c86cf318e8e34435bd1dac117d7cc389 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.193:45489
I20240502 01:54:30.733198 6825 tablet_copy_service.cc:140] P 8214b6c4de6d4aa58a22af551eb8dedd: Received BeginTabletCopySession request for tablet c86cf318e8e34435bd1dac117d7cc389 from peer 3f156b7cfb4f40f481699b257153e5cb ({username='slave'} at 127.31.161.197:46071)
I20240502 01:54:30.733358 6825 tablet_copy_service.cc:161] P 8214b6c4de6d4aa58a22af551eb8dedd: Beginning new tablet copy session on tablet c86cf318e8e34435bd1dac117d7cc389 from peer 3f156b7cfb4f40f481699b257153e5cb at {username='slave'} at 127.31.161.197:46071: session id = 3f156b7cfb4f40f481699b257153e5cb-c86cf318e8e34435bd1dac117d7cc389
I20240502 01:54:30.735306 6825 tablet_copy_source_session.cc:205] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:30.736407 10615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c86cf318e8e34435bd1dac117d7cc389. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:30.740679 10615 tablet_copy_client.cc:783] T c86cf318e8e34435bd1dac117d7cc389 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:30.741024 10615 tablet_copy_client.cc:647] T c86cf318e8e34435bd1dac117d7cc389 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:30.742537 10619 tablet_copy_client.cc:1002] T c86cf318e8e34435bd1dac117d7cc389 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:30.923318 10670 ts_tablet_manager.cc:902] T b984c722354c462882428317c411fd78 P 3f156b7cfb4f40f481699b257153e5cb: Initiating tablet copy from peer fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873)
I20240502 01:54:30.939448 10672 ts_tablet_manager.cc:902] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298: Initiating tablet copy from peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:30.950408 10670 tablet_copy_client.cc:308] T b984c722354c462882428317c411fd78 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:38873
I20240502 01:54:30.951941 7218 tablet_copy_service.cc:140] P fc21155d36d5416fa3650f0268a20b17: Received BeginTabletCopySession request for tablet b984c722354c462882428317c411fd78 from peer 3f156b7cfb4f40f481699b257153e5cb ({username='slave'} at 127.31.161.197:48453)
I20240502 01:54:30.952111 7218 tablet_copy_service.cc:161] P fc21155d36d5416fa3650f0268a20b17: Beginning new tablet copy session on tablet b984c722354c462882428317c411fd78 from peer 3f156b7cfb4f40f481699b257153e5cb at {username='slave'} at 127.31.161.197:48453: session id = 3f156b7cfb4f40f481699b257153e5cb-b984c722354c462882428317c411fd78
I20240502 01:54:30.955024 7218 tablet_copy_source_session.cc:205] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:30.956360 10670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b984c722354c462882428317c411fd78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:30.962194 10670 tablet_copy_client.cc:783] T b984c722354c462882428317c411fd78 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:30.962476 10670 tablet_copy_client.cc:647] T b984c722354c462882428317c411fd78 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:30.968813 10677 tablet_copy_client.cc:1002] T b984c722354c462882428317c411fd78 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:30.970039 10672 tablet_copy_client.cc:308] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.193:45489
I20240502 01:54:30.970954 6825 tablet_copy_service.cc:140] P 8214b6c4de6d4aa58a22af551eb8dedd: Received BeginTabletCopySession request for tablet 4c1d0974a2fd4a548806b6b6560a89f8 from peer 48866680637c40368e45ba844efb9298 ({username='slave'} at 127.31.161.200:42825)
I20240502 01:54:30.971113 6825 tablet_copy_service.cc:161] P 8214b6c4de6d4aa58a22af551eb8dedd: Beginning new tablet copy session on tablet 4c1d0974a2fd4a548806b6b6560a89f8 from peer 48866680637c40368e45ba844efb9298 at {username='slave'} at 127.31.161.200:42825: session id = 48866680637c40368e45ba844efb9298-4c1d0974a2fd4a548806b6b6560a89f8
I20240502 01:54:30.973505 6825 tablet_copy_source_session.cc:205] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:30.974596 10672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c1d0974a2fd4a548806b6b6560a89f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:30.979255 10672 tablet_copy_client.cc:783] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:30.979499 10672 tablet_copy_client.cc:647] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:30.981335 10680 tablet_copy_client.cc:1002] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:30.987403 7197 raft_consensus.cc:1261] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [term 2 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:54:30.987882 10079 consensus_queue.cc:1035] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:31.068763 6801 raft_consensus.cc:1261] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 2 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:54:31.069407 10114 consensus_queue.cc:1035] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:31.073355 8824 raft_consensus.cc:1261] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [term 2 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 01:54:31.074529 10686 consensus_queue.cc:1035] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:31.541386 10349 tablet_copy_client.cc:523] T ee804f1981f84a118051996fb83056b8 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:31.544266 10349 tablet_bootstrap.cc:492] T ee804f1981f84a118051996fb83056b8 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap starting.
I20240502 01:54:31.572947 10363 tablet_copy_client.cc:523] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:31.575898 10363 tablet_bootstrap.cc:492] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298: Bootstrap starting.
I20240502 01:54:31.578526 7067 consensus_queue.cc:237] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:31.580673 6801 raft_consensus.cc:1261] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 2 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:54:31.580714 8824 raft_consensus.cc:1261] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [term 2 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 01:54:31.581455 10112 consensus_queue.cc:1035] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:31.581810 10686 consensus_queue.cc:1035] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:31.584398 10686 raft_consensus.cc:2932] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:31.589469 8824 raft_consensus.cc:2932] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:31.590981 10368 tablet_copy_client.cc:523] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:31.593988 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d58f60c0920e4616850cb246acce1fd7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:31.594480 10368 tablet_bootstrap.cc:492] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap starting.
I20240502 01:54:31.596886 6659 catalog_manager.cc:5484] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index 5 to 7, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New cstate: current_term: 2 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:31.598309 6801 raft_consensus.cc:2932] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:31.614267 10349 tablet_bootstrap.cc:492] T ee804f1981f84a118051996fb83056b8 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:31.614631 10349 tablet_bootstrap.cc:492] T ee804f1981f84a118051996fb83056b8 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap complete.
I20240502 01:54:31.614905 10349 ts_tablet_manager.cc:1367] T ee804f1981f84a118051996fb83056b8 P 9e47e5c098b848c9abe1559cbcbb6321: Time spent bootstrapping tablet: real 0.071s user 0.044s sys 0.000s
I20240502 01:54:31.615765 10349 raft_consensus.cc:348] T ee804f1981f84a118051996fb83056b8 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:31.616048 10349 raft_consensus.cc:729] T ee804f1981f84a118051996fb83056b8 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e47e5c098b848c9abe1559cbcbb6321, State: Initialized, Role: LEARNER
I20240502 01:54:31.616289 10349 consensus_queue.cc:260] T ee804f1981f84a118051996fb83056b8 P 9e47e5c098b848c9abe1559cbcbb6321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:31.616837 10349 ts_tablet_manager.cc:1398] T ee804f1981f84a118051996fb83056b8 P 9e47e5c098b848c9abe1559cbcbb6321: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:31.617619 7218 tablet_copy_service.cc:342] P fc21155d36d5416fa3650f0268a20b17: Request end of tablet copy session 9e47e5c098b848c9abe1559cbcbb6321-ee804f1981f84a118051996fb83056b8 received from {username='slave'} at 127.31.161.198:52833
I20240502 01:54:31.617776 7218 tablet_copy_service.cc:433] P fc21155d36d5416fa3650f0268a20b17: ending tablet copy session 9e47e5c098b848c9abe1559cbcbb6321-ee804f1981f84a118051996fb83056b8 on tablet ee804f1981f84a118051996fb83056b8 with peer 9e47e5c098b848c9abe1559cbcbb6321
I20240502 01:54:31.648348 7179 tablet_service.cc:1514] Processing DeleteTablet for tablet d58f60c0920e4616850cb246acce1fd7 with delete_type TABLET_DATA_TOMBSTONED (TS fc21155d36d5416fa3650f0268a20b17 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:40840
I20240502 01:54:31.650702 10808 tablet_replica.cc:330] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17: stopping tablet replica
I20240502 01:54:31.651041 10808 raft_consensus.cc:2229] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:31.651402 10808 raft_consensus.cc:2258] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:31.653273 10363 tablet_bootstrap.cc:492] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:31.653707 10363 tablet_bootstrap.cc:492] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298: Bootstrap complete.
I20240502 01:54:31.653985 10363 ts_tablet_manager.cc:1367] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298: Time spent bootstrapping tablet: real 0.078s user 0.040s sys 0.004s
I20240502 01:54:31.654309 10808 ts_tablet_manager.cc:1875] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:31.654803 10363 raft_consensus.cc:348] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:31.655092 10363 raft_consensus.cc:729] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 48866680637c40368e45ba844efb9298, State: Initialized, Role: LEARNER
I20240502 01:54:31.655321 10363 consensus_queue.cc:260] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:31.661278 10808 ts_tablet_manager.cc:1888] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 01:54:31.661446 10808 log.cc:1196] T d58f60c0920e4616850cb246acce1fd7 P fc21155d36d5416fa3650f0268a20b17: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/d58f60c0920e4616850cb246acce1fd7
I20240502 01:54:31.662329 6645 catalog_manager.cc:4830] TS fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873): tablet d58f60c0920e4616850cb246acce1fd7 (table rebalance_test_table_7 [id=4d5b7e86d6b24299a377cfaff5756f47]) successfully deleted
I20240502 01:54:31.665726 10363 ts_tablet_manager.cc:1398] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20240502 01:54:31.667305 6958 tablet_copy_service.cc:342] P 137e2183ff57411791e9af58dfab0a3c: Request end of tablet copy session 48866680637c40368e45ba844efb9298-76fb5e1f69284140a6c6da264a124b3e received from {username='slave'} at 127.31.161.200:55333
I20240502 01:54:31.667439 6958 tablet_copy_service.cc:433] P 137e2183ff57411791e9af58dfab0a3c: ending tablet copy session 48866680637c40368e45ba844efb9298-76fb5e1f69284140a6c6da264a124b3e on tablet 76fb5e1f69284140a6c6da264a124b3e with peer 48866680637c40368e45ba844efb9298
I20240502 01:54:31.671844 10368 tablet_bootstrap.cc:492] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:31.672302 10368 tablet_bootstrap.cc:492] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap complete.
I20240502 01:54:31.672605 10368 ts_tablet_manager.cc:1367] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc: Time spent bootstrapping tablet: real 0.078s user 0.050s sys 0.000s
I20240502 01:54:31.673492 10368 raft_consensus.cc:348] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:31.673796 10368 raft_consensus.cc:729] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c39b0639d5fb4d9da09bf1d67d617fcc, State: Initialized, Role: LEARNER
I20240502 01:54:31.674021 10368 consensus_queue.cc:260] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:31.674871 10368 ts_tablet_manager.cc:1398] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:54:31.675725 6958 tablet_copy_service.cc:342] P 137e2183ff57411791e9af58dfab0a3c: Request end of tablet copy session c39b0639d5fb4d9da09bf1d67d617fcc-98630e8d517a4089b236f32f7616cb00 received from {username='slave'} at 127.31.161.199:41907
I20240502 01:54:31.675864 6958 tablet_copy_service.cc:433] P 137e2183ff57411791e9af58dfab0a3c: ending tablet copy session c39b0639d5fb4d9da09bf1d67d617fcc-98630e8d517a4089b236f32f7616cb00 on tablet 98630e8d517a4089b236f32f7616cb00 with peer c39b0639d5fb4d9da09bf1d67d617fcc
I20240502 01:54:31.710089 10423 tablet_copy_client.cc:523] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:31.714208 10423 tablet_bootstrap.cc:492] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298: Bootstrap starting.
I20240502 01:54:31.739626 10431 tablet_copy_client.cc:523] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:31.744859 10431 tablet_bootstrap.cc:492] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap starting.
I20240502 01:54:31.755379 10423 tablet_bootstrap.cc:492] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:31.755833 10423 tablet_bootstrap.cc:492] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298: Bootstrap complete.
I20240502 01:54:31.756171 10423 ts_tablet_manager.cc:1367] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298: Time spent bootstrapping tablet: real 0.042s user 0.029s sys 0.008s
I20240502 01:54:31.757030 10423 raft_consensus.cc:348] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:31.757341 10423 raft_consensus.cc:729] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 48866680637c40368e45ba844efb9298, State: Initialized, Role: LEARNER
I20240502 01:54:31.757558 10423 consensus_queue.cc:260] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:31.758288 10423 ts_tablet_manager.cc:1398] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:54:31.759456 6958 tablet_copy_service.cc:342] P 137e2183ff57411791e9af58dfab0a3c: Request end of tablet copy session 48866680637c40368e45ba844efb9298-8960ca8e5d26429589abc4a154311983 received from {username='slave'} at 127.31.161.200:55333
I20240502 01:54:31.759599 6958 tablet_copy_service.cc:433] P 137e2183ff57411791e9af58dfab0a3c: ending tablet copy session 48866680637c40368e45ba844efb9298-8960ca8e5d26429589abc4a154311983 on tablet 8960ca8e5d26429589abc4a154311983 with peer 48866680637c40368e45ba844efb9298
I20240502 01:54:31.795128 10431 tablet_bootstrap.cc:492] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:31.795528 10431 tablet_bootstrap.cc:492] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap complete.
I20240502 01:54:31.795830 10431 ts_tablet_manager.cc:1367] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc: Time spent bootstrapping tablet: real 0.051s user 0.046s sys 0.000s
I20240502 01:54:31.796571 10431 raft_consensus.cc:348] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:31.796871 10431 raft_consensus.cc:729] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c39b0639d5fb4d9da09bf1d67d617fcc, State: Initialized, Role: LEARNER
I20240502 01:54:31.797135 10431 consensus_queue.cc:260] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:31.797711 10431 ts_tablet_manager.cc:1398] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:54:31.798847 7089 tablet_copy_service.cc:342] P 11c097d7d32a4c22a6385a7465876eb3: Request end of tablet copy session c39b0639d5fb4d9da09bf1d67d617fcc-43e6e5b3435a446499106efe7baf92aa received from {username='slave'} at 127.31.161.199:34537
I20240502 01:54:31.798987 7089 tablet_copy_service.cc:433] P 11c097d7d32a4c22a6385a7465876eb3: ending tablet copy session c39b0639d5fb4d9da09bf1d67d617fcc-43e6e5b3435a446499106efe7baf92aa on tablet 43e6e5b3435a446499106efe7baf92aa with peer c39b0639d5fb4d9da09bf1d67d617fcc
I20240502 01:54:31.818675 8956 raft_consensus.cc:1203] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:31.829916 9086 raft_consensus.cc:1203] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:31.838007 8824 raft_consensus.cc:1203] T ee804f1981f84a118051996fb83056b8 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:31.854043 8956 raft_consensus.cc:1203] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:31.859853 9086 raft_consensus.cc:1203] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:31.904335 10480 tablet_copy_client.cc:523] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:31.907107 10480 tablet_bootstrap.cc:492] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap starting.
I20240502 01:54:31.948642 10480 tablet_bootstrap.cc:492] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:31.949023 10480 tablet_bootstrap.cc:492] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap complete.
I20240502 01:54:31.949294 10480 ts_tablet_manager.cc:1367] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb: Time spent bootstrapping tablet: real 0.042s user 0.036s sys 0.008s
I20240502 01:54:31.950090 10480 raft_consensus.cc:348] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:31.950305 10480 raft_consensus.cc:729] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3f156b7cfb4f40f481699b257153e5cb, State: Initialized, Role: LEARNER
I20240502 01:54:31.950507 10480 consensus_queue.cc:260] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:31.951239 10480 ts_tablet_manager.cc:1398] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:31.952267 7218 tablet_copy_service.cc:342] P fc21155d36d5416fa3650f0268a20b17: Request end of tablet copy session 3f156b7cfb4f40f481699b257153e5cb-70ce4bbee414402aa97c9f3914b9d0c6 received from {username='slave'} at 127.31.161.197:48453
I20240502 01:54:31.952414 7218 tablet_copy_service.cc:433] P fc21155d36d5416fa3650f0268a20b17: ending tablet copy session 3f156b7cfb4f40f481699b257153e5cb-70ce4bbee414402aa97c9f3914b9d0c6 on tablet 70ce4bbee414402aa97c9f3914b9d0c6 with peer 3f156b7cfb4f40f481699b257153e5cb
I20240502 01:54:32.211937 8691 raft_consensus.cc:1203] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:32.219381 10610 tablet_copy_client.cc:523] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:32.222055 10610 tablet_bootstrap.cc:492] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298: Bootstrap starting.
I20240502 01:54:32.242928 10615 tablet_copy_client.cc:523] T c86cf318e8e34435bd1dac117d7cc389 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:32.246632 10615 tablet_bootstrap.cc:492] T c86cf318e8e34435bd1dac117d7cc389 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap starting.
I20240502 01:54:32.258044 10610 tablet_bootstrap.cc:492] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:32.258458 10610 tablet_bootstrap.cc:492] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298: Bootstrap complete.
I20240502 01:54:32.258752 10610 ts_tablet_manager.cc:1367] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298: Time spent bootstrapping tablet: real 0.037s user 0.032s sys 0.004s
I20240502 01:54:32.259541 10610 raft_consensus.cc:348] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:32.259814 10610 raft_consensus.cc:729] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 48866680637c40368e45ba844efb9298, State: Initialized, Role: LEARNER
I20240502 01:54:32.260027 10610 consensus_queue.cc:260] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:32.260618 10610 ts_tablet_manager.cc:1398] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:54:32.261732 7089 tablet_copy_service.cc:342] P 11c097d7d32a4c22a6385a7465876eb3: Request end of tablet copy session 48866680637c40368e45ba844efb9298-640b6474d1834321881094ed0b11b857 received from {username='slave'} at 127.31.161.200:55399
I20240502 01:54:32.261884 7089 tablet_copy_service.cc:433] P 11c097d7d32a4c22a6385a7465876eb3: ending tablet copy session 48866680637c40368e45ba844efb9298-640b6474d1834321881094ed0b11b857 on tablet 640b6474d1834321881094ed0b11b857 with peer 48866680637c40368e45ba844efb9298
I20240502 01:54:32.271857 10924 raft_consensus.cc:1050] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c: attempting to promote NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc to VOTER
I20240502 01:54:32.272859 10924 consensus_queue.cc:237] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:32.275707 8956 raft_consensus.cc:1261] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:32.275835 7067 raft_consensus.cc:1261] T 98630e8d517a4089b236f32f7616cb00 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:32.275892 6801 raft_consensus.cc:1261] T 98630e8d517a4089b236f32f7616cb00 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:32.276366 9395 consensus_queue.cc:1035] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:32.276721 10871 consensus_queue.cc:1035] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:32.277338 9395 consensus_queue.cc:1035] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:32.282572 9395 raft_consensus.cc:2932] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:32.286394 7067 raft_consensus.cc:2932] T 98630e8d517a4089b236f32f7616cb00 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:32.291932 6659 catalog_manager.cc:5484] T 98630e8d517a4089b236f32f7616cb00 P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:32.292661 6801 raft_consensus.cc:2932] T 98630e8d517a4089b236f32f7616cb00 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:32.295308 8956 raft_consensus.cc:2932] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:32.298046 10615 tablet_bootstrap.cc:492] T c86cf318e8e34435bd1dac117d7cc389 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:32.298476 10615 tablet_bootstrap.cc:492] T c86cf318e8e34435bd1dac117d7cc389 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap complete.
I20240502 01:54:32.298782 10615 ts_tablet_manager.cc:1367] T c86cf318e8e34435bd1dac117d7cc389 P 3f156b7cfb4f40f481699b257153e5cb: Time spent bootstrapping tablet: real 0.052s user 0.038s sys 0.008s
I20240502 01:54:32.299897 10615 raft_consensus.cc:348] T c86cf318e8e34435bd1dac117d7cc389 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:32.300148 10615 raft_consensus.cc:729] T c86cf318e8e34435bd1dac117d7cc389 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3f156b7cfb4f40f481699b257153e5cb, State: Initialized, Role: LEARNER
I20240502 01:54:32.300348 10615 consensus_queue.cc:260] T c86cf318e8e34435bd1dac117d7cc389 P 3f156b7cfb4f40f481699b257153e5cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:32.301033 10615 ts_tablet_manager.cc:1398] T c86cf318e8e34435bd1dac117d7cc389 P 3f156b7cfb4f40f481699b257153e5cb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:54:32.305207 6825 tablet_copy_service.cc:342] P 8214b6c4de6d4aa58a22af551eb8dedd: Request end of tablet copy session 3f156b7cfb4f40f481699b257153e5cb-c86cf318e8e34435bd1dac117d7cc389 received from {username='slave'} at 127.31.161.197:46071
I20240502 01:54:32.305351 6825 tablet_copy_service.cc:433] P 8214b6c4de6d4aa58a22af551eb8dedd: ending tablet copy session 3f156b7cfb4f40f481699b257153e5cb-c86cf318e8e34435bd1dac117d7cc389 on tablet c86cf318e8e34435bd1dac117d7cc389 with peer 3f156b7cfb4f40f481699b257153e5cb
I20240502 01:54:32.306596 10245 raft_consensus.cc:1050] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17: attempting to promote NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 to VOTER
I20240502 01:54:32.307441 10245 consensus_queue.cc:237] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:32.316313 8824 raft_consensus.cc:1261] T ee804f1981f84a118051996fb83056b8 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:32.316525 6801 raft_consensus.cc:1261] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:32.317291 10261 consensus_queue.cc:1035] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:32.318006 10261 consensus_queue.cc:1035] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:32.324641 6936 consensus_queue.cc:237] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:32.326861 7067 raft_consensus.cc:1261] T 98630e8d517a4089b236f32f7616cb00 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:32.327275 8956 raft_consensus.cc:1261] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:32.327369 10871 consensus_queue.cc:1035] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:32.327883 10924 consensus_queue.cc:1035] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:32.330296 10871 raft_consensus.cc:2932] T 98630e8d517a4089b236f32f7616cb00 P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:32.333731 8956 raft_consensus.cc:2932] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:32.334713 7067 raft_consensus.cc:2932] T 98630e8d517a4089b236f32f7616cb00 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:32.339080 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 98630e8d517a4089b236f32f7616cb00 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:32.342176 6654 catalog_manager.cc:5484] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc reported cstate change: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:32.347054 6782 tablet_service.cc:1514] Processing DeleteTablet for tablet 98630e8d517a4089b236f32f7616cb00 with delete_type TABLET_DATA_TOMBSTONED (TS 8214b6c4de6d4aa58a22af551eb8dedd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35388
I20240502 01:54:32.347749 7067 raft_consensus.cc:1261] T ee804f1981f84a118051996fb83056b8 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:32.348472 10261 consensus_queue.cc:1035] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:32.361714 10939 tablet_replica.cc:330] T 98630e8d517a4089b236f32f7616cb00 P 8214b6c4de6d4aa58a22af551eb8dedd: stopping tablet replica
I20240502 01:54:32.361876 10934 raft_consensus.cc:2932] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:32.362414 7066 raft_consensus.cc:2932] T ee804f1981f84a118051996fb83056b8 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:32.362653 8824 raft_consensus.cc:2932] T ee804f1981f84a118051996fb83056b8 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:32.366710 10939 raft_consensus.cc:2229] T 98630e8d517a4089b236f32f7616cb00 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:32.367305 10939 raft_consensus.cc:2258] T 98630e8d517a4089b236f32f7616cb00 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:32.367276 6801 raft_consensus.cc:2932] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:32.369971 10939 ts_tablet_manager.cc:1875] T 98630e8d517a4089b236f32f7616cb00 P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:32.370414 6658 catalog_manager.cc:5484] T ee804f1981f84a118051996fb83056b8 P 9e47e5c098b848c9abe1559cbcbb6321 reported cstate change: config changed from index 4 to 5, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:32.376391 7197 consensus_queue.cc:237] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:32.379251 8824 raft_consensus.cc:1261] T ee804f1981f84a118051996fb83056b8 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:32.379841 10261 consensus_queue.cc:1035] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:32.379999 7067 raft_consensus.cc:1261] T ee804f1981f84a118051996fb83056b8 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:32.380609 10261 consensus_queue.cc:1035] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:32.381790 10946 raft_consensus.cc:1050] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3: attempting to promote NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc to VOTER
I20240502 01:54:32.382550 10946 consensus_queue.cc:237] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:32.383620 10261 raft_consensus.cc:2932] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:32.386118 10939 ts_tablet_manager.cc:1888] T 98630e8d517a4089b236f32f7616cb00 P 8214b6c4de6d4aa58a22af551eb8dedd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:32.386253 10939 log.cc:1196] T 98630e8d517a4089b236f32f7616cb00 P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/98630e8d517a4089b236f32f7616cb00
I20240502 01:54:32.387223 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ee804f1981f84a118051996fb83056b8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:32.387480 6645 catalog_manager.cc:4830] TS 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): tablet 98630e8d517a4089b236f32f7616cb00 (table rebalance_test_table_0 [id=145808b6c1a445a19f5a84827cd55658]) successfully deleted
I20240502 01:54:32.388324 8823 raft_consensus.cc:2932] T ee804f1981f84a118051996fb83056b8 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:32.388904 6654 catalog_manager.cc:5484] T ee804f1981f84a118051996fb83056b8 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:32.388334 7067 raft_consensus.cc:2932] T ee804f1981f84a118051996fb83056b8 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:32.395036 6782 tablet_service.cc:1514] Processing DeleteTablet for tablet ee804f1981f84a118051996fb83056b8 with delete_type TABLET_DATA_TOMBSTONED (TS 8214b6c4de6d4aa58a22af551eb8dedd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35388
I20240502 01:54:32.395428 10939 tablet_replica.cc:330] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd: stopping tablet replica
I20240502 01:54:32.395714 10939 raft_consensus.cc:2229] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:32.396085 10939 raft_consensus.cc:2258] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:32.397037 8956 raft_consensus.cc:1261] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:32.398126 10939 ts_tablet_manager.cc:1875] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:32.399547 6801 raft_consensus.cc:1261] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:32.400503 10944 consensus_queue.cc:1035] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:32.400862 10950 consensus_queue.cc:1035] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:32.403095 6936 raft_consensus.cc:1261] T 43e6e5b3435a446499106efe7baf92aa P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:32.404927 10939 ts_tablet_manager.cc:1888] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:32.404892 10946 consensus_queue.cc:1035] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:32.406226 10939 log.cc:1196] T ee804f1981f84a118051996fb83056b8 P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/ee804f1981f84a118051996fb83056b8
I20240502 01:54:32.407068 6645 catalog_manager.cc:4830] TS 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): tablet ee804f1981f84a118051996fb83056b8 (table rebalance_test_table_9 [id=5b2a4358eba544959466c46759744194]) successfully deleted
I20240502 01:54:32.409404 10946 raft_consensus.cc:2932] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:32.410305 6801 raft_consensus.cc:2932] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:32.410353 8956 raft_consensus.cc:2932] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:32.414237 6936 raft_consensus.cc:2932] T 43e6e5b3435a446499106efe7baf92aa P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:32.415904 6654 catalog_manager.cc:5484] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc reported cstate change: config changed from index 4 to 5, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:32.419131 9395 raft_consensus.cc:1050] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c: attempting to promote NON_VOTER 48866680637c40368e45ba844efb9298 to VOTER
I20240502 01:54:32.420042 9395 consensus_queue.cc:237] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:32.425786 7067 consensus_queue.cc:237] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:32.428521 8956 raft_consensus.cc:1261] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:32.429200 10946 consensus_queue.cc:1035] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:32.432401 10950 raft_consensus.cc:2932] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:32.433373 6803 raft_consensus.cc:1261] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:32.438166 8957 raft_consensus.cc:2932] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:32.439134 9086 raft_consensus.cc:1261] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:32.439708 10871 consensus_queue.cc:1035] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:32.441426 10112 consensus_queue.cc:1035] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:32.443024 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 43e6e5b3435a446499106efe7baf92aa with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:32.443024 6801 raft_consensus.cc:2932] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:32.443946 10926 raft_consensus.cc:1050] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c: attempting to promote NON_VOTER 48866680637c40368e45ba844efb9298 to VOTER
I20240502 01:54:32.444756 10926 consensus_queue.cc:237] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:32.447989 6653 catalog_manager.cc:5484] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc reported cstate change: config changed from index 5 to 6, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:32.450634 7067 raft_consensus.cc:1261] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:32.452709 7197 raft_consensus.cc:1261] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:32.453214 6915 tablet_service.cc:1514] Processing DeleteTablet for tablet 43e6e5b3435a446499106efe7baf92aa with delete_type TABLET_DATA_TOMBSTONED (TS 137e2183ff57411791e9af58dfab0a3c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58498
I20240502 01:54:32.453444 9087 raft_consensus.cc:1261] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:32.454130 6801 raft_consensus.cc:1261] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:32.454080 10871 consensus_queue.cc:1035] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:32.454653 10871 consensus_queue.cc:1035] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:32.456601 10924 consensus_queue.cc:1035] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:32.456605 10871 consensus_queue.cc:1035] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:32.457437 7197 raft_consensus.cc:1261] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:32.458527 10924 consensus_queue.cc:1035] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:32.469334 10670 tablet_copy_client.cc:523] T b984c722354c462882428317c411fd78 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:32.473482 10670 tablet_bootstrap.cc:492] T b984c722354c462882428317c411fd78 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap starting.
I20240502 01:54:32.481730 10672 tablet_copy_client.cc:523] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:32.485996 10672 tablet_bootstrap.cc:492] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298: Bootstrap starting.
I20240502 01:54:32.498267 10960 tablet_replica.cc:330] T 43e6e5b3435a446499106efe7baf92aa P 137e2183ff57411791e9af58dfab0a3c: stopping tablet replica
I20240502 01:54:32.498621 10960 raft_consensus.cc:2229] T 43e6e5b3435a446499106efe7baf92aa P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:32.499096 10960 raft_consensus.cc:2258] T 43e6e5b3435a446499106efe7baf92aa P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:32.501150 10926 raft_consensus.cc:2932] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:32.502792 10960 ts_tablet_manager.cc:1875] T 43e6e5b3435a446499106efe7baf92aa P 137e2183ff57411791e9af58dfab0a3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:32.503935 7197 raft_consensus.cc:2932] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:32.505590 7066 raft_consensus.cc:2932] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:32.508822 10871 raft_consensus.cc:2932] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:32.509936 7198 raft_consensus.cc:2932] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:32.513664 6659 catalog_manager.cc:5484] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:32.516073 10960 ts_tablet_manager.cc:1888] T 43e6e5b3435a446499106efe7baf92aa P 137e2183ff57411791e9af58dfab0a3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:32.516252 10960 log.cc:1196] T 43e6e5b3435a446499106efe7baf92aa P 137e2183ff57411791e9af58dfab0a3c: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/wals/43e6e5b3435a446499106efe7baf92aa
I20240502 01:54:32.517701 6646 catalog_manager.cc:4830] TS 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315): tablet 43e6e5b3435a446499106efe7baf92aa (table rebalance_test_table_7 [id=4d5b7e86d6b24299a377cfaff5756f47]) successfully deleted
I20240502 01:54:32.520231 6659 catalog_manager.cc:5484] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:32.522190 9087 raft_consensus.cc:2932] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:32.526822 9086 raft_consensus.cc:2932] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:32.531517 6936 consensus_queue.cc:237] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:32.533979 7198 raft_consensus.cc:1261] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:32.534557 10926 consensus_queue.cc:1035] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:32.536906 6936 consensus_queue.cc:237] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:32.537215 10926 raft_consensus.cc:2932] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:32.538789 9087 raft_consensus.cc:1261] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:32.540409 10924 consensus_queue.cc:1035] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:32.541870 9085 raft_consensus.cc:2932] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:32.542523 9087 raft_consensus.cc:1261] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:32.543309 7197 raft_consensus.cc:2932] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:32.543381 6801 raft_consensus.cc:2932] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:32.546142 10926 consensus_queue.cc:1035] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:32.546373 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 76fb5e1f69284140a6c6da264a124b3e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:32.549942 6803 raft_consensus.cc:1261] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:32.551558 9395 raft_consensus.cc:2932] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:32.554996 6658 catalog_manager.cc:5484] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298 reported cstate change: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:32.556233 10871 consensus_queue.cc:1035] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:32.557888 6803 raft_consensus.cc:2932] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:32.561538 7047 tablet_service.cc:1514] Processing DeleteTablet for tablet 76fb5e1f69284140a6c6da264a124b3e with delete_type TABLET_DATA_TOMBSTONED (TS 11c097d7d32a4c22a6385a7465876eb3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59762
I20240502 01:54:32.561820 9086 raft_consensus.cc:2932] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:32.562427 10974 tablet_replica.cc:330] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3: stopping tablet replica
I20240502 01:54:32.562709 10974 raft_consensus.cc:2229] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:32.563040 10974 raft_consensus.cc:2258] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:32.564066 10974 ts_tablet_manager.cc:1875] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:32.566493 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8960ca8e5d26429589abc4a154311983 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:32.569799 6654 catalog_manager.cc:5484] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c reported cstate change: config changed from index 5 to 6, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:32.571519 10974 ts_tablet_manager.cc:1888] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:32.571702 10974 log.cc:1196] T 76fb5e1f69284140a6c6da264a124b3e P 11c097d7d32a4c22a6385a7465876eb3: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/wals/76fb5e1f69284140a6c6da264a124b3e
I20240502 01:54:32.572602 6646 catalog_manager.cc:4830] TS 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083): tablet 76fb5e1f69284140a6c6da264a124b3e (table rebalance_test_table_5 [id=abae7d7cdd254582beebaba1abe05ac1]) successfully deleted
I20240502 01:54:32.573940 7179 tablet_service.cc:1514] Processing DeleteTablet for tablet 8960ca8e5d26429589abc4a154311983 with delete_type TABLET_DATA_TOMBSTONED (TS fc21155d36d5416fa3650f0268a20b17 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:40840
I20240502 01:54:32.585729 10975 tablet_replica.cc:330] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17: stopping tablet replica
I20240502 01:54:32.586055 10975 raft_consensus.cc:2229] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:32.586422 10975 raft_consensus.cc:2258] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:32.587627 10975 ts_tablet_manager.cc:1875] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:32.589231 10672 tablet_bootstrap.cc:492] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:32.589421 10670 tablet_bootstrap.cc:492] T b984c722354c462882428317c411fd78 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:32.589890 10670 tablet_bootstrap.cc:492] T b984c722354c462882428317c411fd78 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap complete.
I20240502 01:54:32.589897 10672 tablet_bootstrap.cc:492] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298: Bootstrap complete.
I20240502 01:54:32.590164 10672 ts_tablet_manager.cc:1367] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298: Time spent bootstrapping tablet: real 0.104s user 0.044s sys 0.005s
I20240502 01:54:32.590282 10670 ts_tablet_manager.cc:1367] T b984c722354c462882428317c411fd78 P 3f156b7cfb4f40f481699b257153e5cb: Time spent bootstrapping tablet: real 0.117s user 0.046s sys 0.000s
I20240502 01:54:32.590945 10672 raft_consensus.cc:348] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:32.591009 10670 raft_consensus.cc:348] T b984c722354c462882428317c411fd78 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:32.591202 10672 raft_consensus.cc:729] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 48866680637c40368e45ba844efb9298, State: Initialized, Role: LEARNER
I20240502 01:54:32.591246 10670 raft_consensus.cc:729] T b984c722354c462882428317c411fd78 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3f156b7cfb4f40f481699b257153e5cb, State: Initialized, Role: LEARNER
I20240502 01:54:32.591392 10672 consensus_queue.cc:260] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:32.591424 10670 consensus_queue.cc:260] T b984c722354c462882428317c411fd78 P 3f156b7cfb4f40f481699b257153e5cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:32.592051 10672 ts_tablet_manager.cc:1398] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:32.592804 6825 tablet_copy_service.cc:342] P 8214b6c4de6d4aa58a22af551eb8dedd: Request end of tablet copy session 48866680637c40368e45ba844efb9298-4c1d0974a2fd4a548806b6b6560a89f8 received from {username='slave'} at 127.31.161.200:42825
I20240502 01:54:32.592936 6825 tablet_copy_service.cc:433] P 8214b6c4de6d4aa58a22af551eb8dedd: ending tablet copy session 48866680637c40368e45ba844efb9298-4c1d0974a2fd4a548806b6b6560a89f8 on tablet 4c1d0974a2fd4a548806b6b6560a89f8 with peer 48866680637c40368e45ba844efb9298
I20240502 01:54:32.593420 10670 ts_tablet_manager.cc:1398] T b984c722354c462882428317c411fd78 P 3f156b7cfb4f40f481699b257153e5cb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:32.594434 7218 tablet_copy_service.cc:342] P fc21155d36d5416fa3650f0268a20b17: Request end of tablet copy session 3f156b7cfb4f40f481699b257153e5cb-b984c722354c462882428317c411fd78 received from {username='slave'} at 127.31.161.197:48453
I20240502 01:54:32.594560 7218 tablet_copy_service.cc:433] P fc21155d36d5416fa3650f0268a20b17: ending tablet copy session 3f156b7cfb4f40f481699b257153e5cb-b984c722354c462882428317c411fd78 on tablet b984c722354c462882428317c411fd78 with peer 3f156b7cfb4f40f481699b257153e5cb
I20240502 01:54:32.598696 10975 ts_tablet_manager.cc:1888] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:32.598812 10975 log.cc:1196] T 8960ca8e5d26429589abc4a154311983 P fc21155d36d5416fa3650f0268a20b17: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/8960ca8e5d26429589abc4a154311983
I20240502 01:54:32.599480 6645 catalog_manager.cc:4830] TS fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873): tablet 8960ca8e5d26429589abc4a154311983 (table rebalance_test_table_6 [id=981b993f808c4320845154760fb7959f]) successfully deleted
I20240502 01:54:32.702728 10245 raft_consensus.cc:1050] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17: attempting to promote NON_VOTER 3f156b7cfb4f40f481699b257153e5cb to VOTER
I20240502 01:54:32.703473 10245 consensus_queue.cc:237] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:32.706566 8691 raft_consensus.cc:1261] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:32.706974 6936 raft_consensus.cc:1261] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:32.707587 7066 raft_consensus.cc:1261] T 70ce4bbee414402aa97c9f3914b9d0c6 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:32.708755 10934 consensus_queue.cc:1035] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:32.709563 10934 consensus_queue.cc:1035] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:32.710316 10261 consensus_queue.cc:1035] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:32.729934 10934 raft_consensus.cc:2932] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:32.731971 8691 raft_consensus.cc:2932] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:32.733830 7066 raft_consensus.cc:2932] T 70ce4bbee414402aa97c9f3914b9d0c6 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:32.734920 6936 raft_consensus.cc:2932] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:32.736265 6654 catalog_manager.cc:5484] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:32.743007 7197 consensus_queue.cc:237] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:32.745116 6936 raft_consensus.cc:1261] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:32.745396 8691 raft_consensus.cc:1261] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:32.745895 11028 consensus_queue.cc:1035] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:32.746694 10245 consensus_queue.cc:1035] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:32.748620 10245 raft_consensus.cc:2932] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:32.749722 6936 raft_consensus.cc:2932] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:32.749814 8691 raft_consensus.cc:2932] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:32.756103 6659 catalog_manager.cc:5484] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb reported cstate change: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:32.758101 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 70ce4bbee414402aa97c9f3914b9d0c6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:32.761547 7047 tablet_service.cc:1514] Processing DeleteTablet for tablet 70ce4bbee414402aa97c9f3914b9d0c6 with delete_type TABLET_DATA_TOMBSTONED (TS 11c097d7d32a4c22a6385a7465876eb3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59762
I20240502 01:54:32.761862 10974 tablet_replica.cc:330] T 70ce4bbee414402aa97c9f3914b9d0c6 P 11c097d7d32a4c22a6385a7465876eb3: stopping tablet replica
I20240502 01:54:32.762035 10974 raft_consensus.cc:2229] T 70ce4bbee414402aa97c9f3914b9d0c6 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:32.762310 10974 raft_consensus.cc:2258] T 70ce4bbee414402aa97c9f3914b9d0c6 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:32.763219 10974 ts_tablet_manager.cc:1875] T 70ce4bbee414402aa97c9f3914b9d0c6 P 11c097d7d32a4c22a6385a7465876eb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:32.767199 8691 raft_consensus.cc:1203] T c86cf318e8e34435bd1dac117d7cc389 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:32.772640 10974 ts_tablet_manager.cc:1888] T 70ce4bbee414402aa97c9f3914b9d0c6 P 11c097d7d32a4c22a6385a7465876eb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:32.772763 10974 log.cc:1196] T 70ce4bbee414402aa97c9f3914b9d0c6 P 11c097d7d32a4c22a6385a7465876eb3: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/wals/70ce4bbee414402aa97c9f3914b9d0c6
I20240502 01:54:32.773427 6646 catalog_manager.cc:4830] TS 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083): tablet 70ce4bbee414402aa97c9f3914b9d0c6 (table rebalance_test_table_1 [id=9a08c31578ae4224915289c7c289ee7a]) successfully deleted
I20240502 01:54:32.811210 9086 raft_consensus.cc:1203] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:32.915421 9086 raft_consensus.cc:1203] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:32.948458 8691 raft_consensus.cc:1203] T b984c722354c462882428317c411fd78 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:54:33.293300 11103 raft_consensus.cc:1050] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd: attempting to promote NON_VOTER 3f156b7cfb4f40f481699b257153e5cb to VOTER
I20240502 01:54:33.294145 10950 raft_consensus.cc:1050] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3: attempting to promote NON_VOTER 48866680637c40368e45ba844efb9298 to VOTER
I20240502 01:54:33.294997 10950 consensus_queue.cc:237] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:33.297592 6801 raft_consensus.cc:1261] T 640b6474d1834321881094ed0b11b857 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:33.298211 7197 raft_consensus.cc:1261] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:33.298842 11062 consensus_queue.cc:1035] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:33.299268 10946 consensus_queue.cc:1035] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:33.300479 9086 raft_consensus.cc:1261] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:33.300958 11062 consensus_queue.cc:1035] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:33.312165 11103 consensus_queue.cc:237] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:33.320175 8691 raft_consensus.cc:1261] T c86cf318e8e34435bd1dac117d7cc389 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:33.320657 6936 raft_consensus.cc:1261] T c86cf318e8e34435bd1dac117d7cc389 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:33.320705 10946 raft_consensus.cc:2932] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:33.321287 7066 raft_consensus.cc:1261] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:33.322203 8246 consensus_queue.cc:1035] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:33.322820 8246 consensus_queue.cc:1035] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:33.323280 8246 consensus_queue.cc:1035] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:33.325176 9085 raft_consensus.cc:2932] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:33.330961 6654 catalog_manager.cc:5484] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298 reported cstate change: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:33.339347 7066 raft_consensus.cc:2932] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:33.339592 7197 raft_consensus.cc:2932] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:33.341842 6801 raft_consensus.cc:2932] T 640b6474d1834321881094ed0b11b857 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:33.345559 7067 consensus_queue.cc:237] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:33.338640 11111 raft_consensus.cc:2932] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:33.347723 9086 raft_consensus.cc:1261] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:33.348757 11062 consensus_queue.cc:1035] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:33.350394 6935 raft_consensus.cc:2932] T c86cf318e8e34435bd1dac117d7cc389 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:33.351377 11062 raft_consensus.cc:2932] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:33.352671 6658 catalog_manager.cc:5484] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:33.353506 11104 raft_consensus.cc:1050] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd: attempting to promote NON_VOTER 48866680637c40368e45ba844efb9298 to VOTER
I20240502 01:54:33.354279 11104 consensus_queue.cc:237] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:33.355036 7197 raft_consensus.cc:1261] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:33.355820 9086 raft_consensus.cc:2932] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:33.358309 9085 raft_consensus.cc:1261] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:33.358412 6935 raft_consensus.cc:1261] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:33.358840 8246 consensus_queue.cc:1035] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:33.359200 11103 consensus_queue.cc:1035] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:33.359529 10946 consensus_queue.cc:1035] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:33.361065 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 640b6474d1834321881094ed0b11b857 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:33.363425 7066 raft_consensus.cc:1261] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:33.363940 8246 consensus_queue.cc:1035] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:33.363775 7197 raft_consensus.cc:2932] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:33.365377 6659 catalog_manager.cc:5484] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298 reported cstate change: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:33.368172 8691 raft_consensus.cc:2932] T c86cf318e8e34435bd1dac117d7cc389 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:33.385614 6782 tablet_service.cc:1514] Processing DeleteTablet for tablet 640b6474d1834321881094ed0b11b857 with delete_type TABLET_DATA_TOMBSTONED (TS 8214b6c4de6d4aa58a22af551eb8dedd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35388
I20240502 01:54:33.388376 11125 tablet_replica.cc:330] T 640b6474d1834321881094ed0b11b857 P 8214b6c4de6d4aa58a22af551eb8dedd: stopping tablet replica
I20240502 01:54:33.388701 11125 raft_consensus.cc:2229] T 640b6474d1834321881094ed0b11b857 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:33.389170 11125 raft_consensus.cc:2258] T 640b6474d1834321881094ed0b11b857 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:33.390290 11103 raft_consensus.cc:2932] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:33.392496 6935 raft_consensus.cc:2932] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:33.395226 6801 consensus_queue.cc:237] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:33.397619 6935 raft_consensus.cc:1261] T c86cf318e8e34435bd1dac117d7cc389 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:33.397619 6654 catalog_manager.cc:5484] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c reported cstate change: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:33.398283 11104 consensus_queue.cc:1035] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:33.398772 10261 raft_consensus.cc:1050] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17: attempting to promote NON_VOTER 3f156b7cfb4f40f481699b257153e5cb to VOTER
I20240502 01:54:33.399103 8691 raft_consensus.cc:1261] T c86cf318e8e34435bd1dac117d7cc389 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:33.399690 11119 consensus_queue.cc:1035] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:33.399637 10261 consensus_queue.cc:237] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:33.402144 8246 raft_consensus.cc:2932] T c86cf318e8e34435bd1dac117d7cc389 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:33.405776 9086 raft_consensus.cc:2932] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:33.406042 7067 raft_consensus.cc:2932] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:33.409010 6936 raft_consensus.cc:2932] T c86cf318e8e34435bd1dac117d7cc389 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:33.411031 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c86cf318e8e34435bd1dac117d7cc389 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:33.411538 8691 raft_consensus.cc:2932] T c86cf318e8e34435bd1dac117d7cc389 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:33.412523 11125 ts_tablet_manager.cc:1875] T 640b6474d1834321881094ed0b11b857 P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:33.413105 8692 raft_consensus.cc:1261] T b984c722354c462882428317c411fd78 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:33.413892 10245 consensus_queue.cc:1035] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:33.415620 7066 raft_consensus.cc:1261] T b984c722354c462882428317c411fd78 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:33.417534 6801 raft_consensus.cc:1261] T b984c722354c462882428317c411fd78 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:54:33.418454 10261 consensus_queue.cc:1035] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:33.418581 6659 catalog_manager.cc:5484] T c86cf318e8e34435bd1dac117d7cc389 P 3f156b7cfb4f40f481699b257153e5cb reported cstate change: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:33.425328 7047 tablet_service.cc:1514] Processing DeleteTablet for tablet c86cf318e8e34435bd1dac117d7cc389 with delete_type TABLET_DATA_TOMBSTONED (TS 11c097d7d32a4c22a6385a7465876eb3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59762
I20240502 01:54:33.426839 11125 ts_tablet_manager.cc:1888] T 640b6474d1834321881094ed0b11b857 P 8214b6c4de6d4aa58a22af551eb8dedd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:33.427163 11125 log.cc:1196] T 640b6474d1834321881094ed0b11b857 P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/640b6474d1834321881094ed0b11b857
I20240502 01:54:33.427664 6801 consensus_queue.cc:237] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:33.428251 6645 catalog_manager.cc:4830] TS 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): tablet 640b6474d1834321881094ed0b11b857 (table rebalance_test_table_4 [id=9231a7576c804559bc4a4c54496d5223]) successfully deleted
I20240502 01:54:33.430364 10245 consensus_queue.cc:1035] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:54:33.431689 6936 raft_consensus.cc:1261] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:33.432302 11103 consensus_queue.cc:1035] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:33.432734 9086 raft_consensus.cc:1261] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:33.433444 11103 consensus_queue.cc:1035] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:33.435024 11104 raft_consensus.cc:2932] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:33.436075 6936 raft_consensus.cc:2932] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:33.437258 9085 raft_consensus.cc:2932] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:33.439036 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4c1d0974a2fd4a548806b6b6560a89f8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:33.441895 6654 catalog_manager.cc:5484] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index 5 to 6, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:33.442667 11132 tablet_replica.cc:330] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3: stopping tablet replica
I20240502 01:54:33.443499 11132 raft_consensus.cc:2229] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:33.445695 7047 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c1d0974a2fd4a548806b6b6560a89f8 with delete_type TABLET_DATA_TOMBSTONED (TS 11c097d7d32a4c22a6385a7465876eb3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59762
I20240502 01:54:33.445763 11132 raft_consensus.cc:2258] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:33.448326 10261 raft_consensus.cc:2932] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:33.449052 7067 raft_consensus.cc:2932] T b984c722354c462882428317c411fd78 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:33.452702 6801 raft_consensus.cc:2932] T b984c722354c462882428317c411fd78 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:33.453284 11132 ts_tablet_manager.cc:1875] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:33.453305 6654 catalog_manager.cc:5484] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:33.460237 8691 raft_consensus.cc:2932] T b984c722354c462882428317c411fd78 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:33.461910 7197 consensus_queue.cc:237] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:33.464772 8691 raft_consensus.cc:1261] T b984c722354c462882428317c411fd78 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:33.465057 7067 raft_consensus.cc:1261] T b984c722354c462882428317c411fd78 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:54:33.465479 11129 consensus_queue.cc:1035] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:33.466739 10261 consensus_queue.cc:1035] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:54:33.469059 11132 ts_tablet_manager.cc:1888] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:33.469218 11132 log.cc:1196] T c86cf318e8e34435bd1dac117d7cc389 P 11c097d7d32a4c22a6385a7465876eb3: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/wals/c86cf318e8e34435bd1dac117d7cc389
I20240502 01:54:33.469599 10261 raft_consensus.cc:2932] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:33.470293 6646 catalog_manager.cc:4830] TS 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083): tablet c86cf318e8e34435bd1dac117d7cc389 (table rebalance_test_table_8 [id=8abdf44bc9e2414b9e4f80bf4c87c953]) successfully deleted
I20240502 01:54:33.470779 11132 tablet_replica.cc:330] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3: stopping tablet replica
I20240502 01:54:33.470620 7067 raft_consensus.cc:2932] T b984c722354c462882428317c411fd78 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:33.470789 8691 raft_consensus.cc:2932] T b984c722354c462882428317c411fd78 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:33.471318 11132 raft_consensus.cc:2229] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:33.471686 11132 raft_consensus.cc:2258] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:33.472867 11132 ts_tablet_manager.cc:1875] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:33.474027 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b984c722354c462882428317c411fd78 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:33.480537 11132 ts_tablet_manager.cc:1888] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:33.480667 11132 log.cc:1196] T 4c1d0974a2fd4a548806b6b6560a89f8 P 11c097d7d32a4c22a6385a7465876eb3: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/wals/4c1d0974a2fd4a548806b6b6560a89f8
I20240502 01:54:33.481348 6646 catalog_manager.cc:4830] TS 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083): tablet 4c1d0974a2fd4a548806b6b6560a89f8 (table rebalance_test_table_9 [id=5b2a4358eba544959466c46759744194]) successfully deleted
I20240502 01:54:33.476766 6654 catalog_manager.cc:5484] T b984c722354c462882428317c411fd78 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 5 to 6, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:33.484802 6782 tablet_service.cc:1514] Processing DeleteTablet for tablet b984c722354c462882428317c411fd78 with delete_type TABLET_DATA_TOMBSTONED (TS 8214b6c4de6d4aa58a22af551eb8dedd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:35388
I20240502 01:54:33.485996 11125 tablet_replica.cc:330] T b984c722354c462882428317c411fd78 P 8214b6c4de6d4aa58a22af551eb8dedd: stopping tablet replica
I20240502 01:54:33.486217 11125 raft_consensus.cc:2229] T b984c722354c462882428317c411fd78 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:33.486563 11125 raft_consensus.cc:2258] T b984c722354c462882428317c411fd78 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:33.487548 11125 ts_tablet_manager.cc:1875] T b984c722354c462882428317c411fd78 P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:33.492151 11125 ts_tablet_manager.cc:1888] T b984c722354c462882428317c411fd78 P 8214b6c4de6d4aa58a22af551eb8dedd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:54:33.492273 11125 log.cc:1196] T b984c722354c462882428317c411fd78 P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/b984c722354c462882428317c411fd78
I20240502 01:54:33.492954 6645 catalog_manager.cc:4830] TS 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): tablet b984c722354c462882428317c411fd78 (table rebalance_test_table_2 [id=3316a56a8d6043c6b05160d6e55a33e0]) successfully deleted
I20240502 01:54:33.747026 7047 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:33.806872 6782 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:33.812734 6915 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:33.822952 9067 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:33.835163 8672 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:33.837977 8804 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:33.853893 7179 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:33.905966 8937 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:33.922917 9988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:34.022207 7197 consensus_queue.cc:237] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:34.024680 8823 raft_consensus.cc:1261] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:34.025559 11212 consensus_queue.cc:1035] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.025843 6936 raft_consensus.cc:1261] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:34.026484 11212 consensus_queue.cc:1035] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.030768 11129 raft_consensus.cc:2932] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:34.032172 6936 raft_consensus.cc:2932] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
W20240502 01:54:34.032604 7134 consensus_peers.cc:485] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 -> Peer fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201:43211): Couldn't send request to peer fdfc60a29ea9465c88ee45c3c332caa1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5acc7c2405464a328b2016bb2293f7e8. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:34.033958 8823 raft_consensus.cc:2932] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:34.036753 6659 catalog_manager.cc:5484] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:34.051349 7197 consensus_queue.cc:237] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:34.053480 6936 raft_consensus.cc:1261] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:34.053647 8691 raft_consensus.cc:1261] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:54:34.053747 7135 consensus_peers.cc:485] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 -> Peer 48866680637c40368e45ba844efb9298 (127.31.161.200:34863): Couldn't send request to peer 48866680637c40368e45ba844efb9298. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70ce4bbee414402aa97c9f3914b9d0c6. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:34.054095 11129 consensus_queue.cc:1035] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.054368 10261 consensus_queue.cc:1035] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.056828 11212 raft_consensus.cc:2932] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:34.057518 8691 raft_consensus.cc:2932] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:34.057595 6936 raft_consensus.cc:2932] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } } }
I20240502 01:54:34.062152 6659 catalog_manager.cc:5484] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 6 to 7, NON_VOTER 48866680637c40368e45ba844efb9298 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:34.074442 6936 consensus_queue.cc:237] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
W20240502 01:54:34.077425 6871 consensus_peers.cc:485] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c -> Peer 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198:37123): Couldn't send request to peer 9e47e5c098b848c9abe1559cbcbb6321. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db7a48f131f141f0a4e45036434f8129. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:34.077723 6801 raft_consensus.cc:1261] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:34.077721 8691 raft_consensus.cc:1261] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:34.078250 10871 consensus_queue.cc:1035] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.078573 11235 consensus_queue.cc:1035] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.081432 10871 raft_consensus.cc:2932] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:34.082051 8691 raft_consensus.cc:2932] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:34.087852 6654 catalog_manager.cc:5484] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb reported cstate change: config changed from index 6 to 7, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:34.090795 6801 raft_consensus.cc:2932] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:34.109293 7067 consensus_queue.cc:237] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:34.111726 6801 raft_consensus.cc:1261] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:34.112277 9085 raft_consensus.cc:1261] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:34.112818 11245 consensus_queue.cc:1035] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 01:54:34.113199 7003 consensus_peers.cc:485] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 -> Peer c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199:35245): Couldn't send request to peer c39b0639d5fb4d9da09bf1d67d617fcc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6a165cbc26f4defb70866779f83bfdc. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:34.113215 11205 consensus_queue.cc:1035] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.115839 11106 raft_consensus.cc:2932] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:34.116396 6801 raft_consensus.cc:2932] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:34.117358 9085 raft_consensus.cc:2932] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:34.122323 6659 catalog_manager.cc:5484] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index 6 to 7, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:34.134579 7067 consensus_queue.cc:237] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:34.136425 9085 raft_consensus.cc:1261] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:54:34.136624 7003 consensus_peers.cc:485] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 -> Peer 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197:44451): Couldn't send request to peer 3f156b7cfb4f40f481699b257153e5cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 640b6474d1834321881094ed0b11b857. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:34.136765 7197 raft_consensus.cc:1261] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:34.136967 11106 consensus_queue.cc:1035] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.137512 11106 consensus_queue.cc:1035] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.139142 11106 raft_consensus.cc:2932] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:34.139799 9085 raft_consensus.cc:2932] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:34.140323 7197 raft_consensus.cc:2932] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:34.146834 6653 catalog_manager.cc:5484] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 6 to 7, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:34.165123 6936 consensus_queue.cc:237] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:34.167794 6801 raft_consensus.cc:1261] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:34.168025 9085 raft_consensus.cc:1261] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:34.168447 11235 consensus_queue.cc:1035] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.168766 11238 consensus_queue.cc:1035] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.174870 10871 raft_consensus.cc:2932] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:34.175827 9085 raft_consensus.cc:2932] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:34.176671 6801 raft_consensus.cc:2932] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:34.181089 6654 catalog_manager.cc:5484] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c reported cstate change: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:54:34.185233 6871 consensus_peers.cc:485] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c -> Peer fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201:43211): Couldn't send request to peer fdfc60a29ea9465c88ee45c3c332caa1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8960ca8e5d26429589abc4a154311983. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:34.197948 6936 consensus_queue.cc:237] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:34.200264 9085 raft_consensus.cc:1261] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:34.200939 11238 consensus_queue.cc:1035] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.201187 7197 raft_consensus.cc:1261] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:34.201871 10871 consensus_queue.cc:1035] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 01:54:34.202173 6871 consensus_peers.cc:485] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c -> Peer fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201:43211): Couldn't send request to peer fdfc60a29ea9465c88ee45c3c332caa1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dea582c2912241609405695eed436abe. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:34.203727 11238 raft_consensus.cc:2932] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:34.204396 9085 raft_consensus.cc:2932] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:34.208390 7197 raft_consensus.cc:2932] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:34.209024 6654 catalog_manager.cc:5484] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298 reported cstate change: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:34.226372 7197 consensus_queue.cc:237] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:34.228870 9085 raft_consensus.cc:1261] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:54:34.228961 7134 consensus_peers.cc:485] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 -> Peer c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199:35245): Couldn't send request to peer c39b0639d5fb4d9da09bf1d67d617fcc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d8c4a55e09d41e391b8ba318dfe6896. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:34.229202 6801 raft_consensus.cc:1261] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:34.229450 10261 consensus_queue.cc:1035] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.229755 11212 consensus_queue.cc:1035] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.232267 11212 raft_consensus.cc:2932] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:34.232985 9085 raft_consensus.cc:2932] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:34.235803 6801 raft_consensus.cc:2932] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:34.238224 6659 catalog_manager.cc:5484] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298 reported cstate change: config changed from index 6 to 7, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:34.254582 7197 consensus_queue.cc:237] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:34.256847 8691 raft_consensus.cc:1261] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:34.257975 11280 consensus_queue.cc:1035] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.258255 7067 raft_consensus.cc:1261] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:54:34.258544 7134 consensus_peers.cc:485] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 -> Peer 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198:37123): Couldn't send request to peer 9e47e5c098b848c9abe1559cbcbb6321. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d55da66aad2140d788704b056e476e99. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:34.258847 11212 consensus_queue.cc:1035] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.262039 10261 raft_consensus.cc:2932] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:34.262620 7067 raft_consensus.cc:2932] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:34.262713 8691 raft_consensus.cc:2932] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:34.267740 6654 catalog_manager.cc:5484] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 6 to 7, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:34.282898 7067 consensus_queue.cc:237] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:34.285238 6801 raft_consensus.cc:1261] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 2 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:54:34.285504 8824 raft_consensus.cc:1261] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [term 2 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 01:54:34.285820 11205 consensus_queue.cc:1035] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:34.286096 11245 consensus_queue.cc:1035] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 01:54:34.286484 7003 consensus_peers.cc:485] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 -> Peer 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197:44451): Couldn't send request to peer 3f156b7cfb4f40f481699b257153e5cb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d58f60c0920e4616850cb246acce1fd7. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:34.289338 11106 raft_consensus.cc:2932] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:34.289888 6801 raft_consensus.cc:2932] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:34.290212 8824 raft_consensus.cc:2932] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:34.294924 6653 catalog_manager.cc:5484] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 reported cstate change: config changed from index 7 to 8, NON_VOTER 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) added. New cstate: current_term: 2 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } } }
I20240502 01:54:34.319211 6801 consensus_queue.cc:237] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:34.321843 8957 raft_consensus.cc:1261] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:34.321902 6936 raft_consensus.cc:1261] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:54:34.322304 6738 consensus_peers.cc:485] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd -> Peer 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198:37123): Couldn't send request to peer 9e47e5c098b848c9abe1559cbcbb6321. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f03b37fd2f844f61a39daadc698c9b29. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:34.322651 11103 consensus_queue.cc:1035] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.323024 11128 consensus_queue.cc:1035] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.325829 11103 raft_consensus.cc:2932] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:34.330101 8957 raft_consensus.cc:2932] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:34.336386 6653 catalog_manager.cc:5484] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index 6 to 7, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:34.342192 6936 raft_consensus.cc:2932] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:34.360770 6936 consensus_queue.cc:237] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:34.363126 9085 raft_consensus.cc:1261] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:34.363132 7197 raft_consensus.cc:1261] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:54:34.363370 6871 consensus_peers.cc:485] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c -> Peer fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201:43211): Couldn't send request to peer fdfc60a29ea9465c88ee45c3c332caa1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76fb5e1f69284140a6c6da264a124b3e. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:34.363772 11235 consensus_queue.cc:1035] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.364063 10871 consensus_queue.cc:1035] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.366770 11238 raft_consensus.cc:2932] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:34.367457 9085 raft_consensus.cc:2932] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:34.367395 7197 raft_consensus.cc:2932] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:34.372649 6653 catalog_manager.cc:5484] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298 reported cstate change: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:34.387655 6801 consensus_queue.cc:237] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
W20240502 01:54:34.390049 6738 consensus_peers.cc:485] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd -> Peer c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199:35245): Couldn't send request to peer c39b0639d5fb4d9da09bf1d67d617fcc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c1d0974a2fd4a548806b6b6560a89f8. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:34.390166 9085 raft_consensus.cc:1261] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:34.390441 6936 raft_consensus.cc:1261] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:34.390740 11304 consensus_queue.cc:1035] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.391083 11128 consensus_queue.cc:1035] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.393453 11103 raft_consensus.cc:2932] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:34.394054 9085 raft_consensus.cc:2932] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:34.394199 6936 raft_consensus.cc:2932] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:34.398793 6659 catalog_manager.cc:5484] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298 reported cstate change: config changed from index 6 to 7, NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) added. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } } }
I20240502 01:54:34.412653 6801 consensus_queue.cc:237] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:34.415148 6936 raft_consensus.cc:1261] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:34.415189 8957 raft_consensus.cc:1261] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:54:34.415509 6738 consensus_peers.cc:485] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd -> Peer 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198:37123): Couldn't send request to peer 9e47e5c098b848c9abe1559cbcbb6321. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1f4c94a105741339ba69e641244ac18. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:34.415803 11304 consensus_queue.cc:1035] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.416086 11103 consensus_queue.cc:1035] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.418853 11323 raft_consensus.cc:2932] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:34.419525 6936 raft_consensus.cc:2932] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:34.419662 8957 raft_consensus.cc:2932] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } } }
I20240502 01:54:34.423604 6653 catalog_manager.cc:5484] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index 6 to 7, NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) added. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:34.438639 6936 consensus_queue.cc:237] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:34.440920 8824 raft_consensus.cc:1261] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:34.441154 7197 raft_consensus.cc:1261] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:34.441686 11238 consensus_queue.cc:1035] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 01:54:34.441973 6871 consensus_peers.cc:485] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c -> Peer fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201:43211): Couldn't send request to peer fdfc60a29ea9465c88ee45c3c332caa1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01e98643312b4a2eb80e3bcb3d95674c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:34.442205 11238 consensus_queue.cc:1035] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:34.444594 11235 raft_consensus.cc:2932] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:34.449674 8824 raft_consensus.cc:2932] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:34.453608 11338 ts_tablet_manager.cc:902] T 70ce4bbee414402aa97c9f3914b9d0c6 P 48866680637c40368e45ba844efb9298: Initiating tablet copy from peer fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873)
I20240502 01:54:34.455224 7197 raft_consensus.cc:2932] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:34.459669 6657 catalog_manager.cc:5484] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c reported cstate change: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:34.472147 11338 tablet_copy_client.cc:308] T 70ce4bbee414402aa97c9f3914b9d0c6 P 48866680637c40368e45ba844efb9298: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:38873
I20240502 01:54:34.472895 7218 tablet_copy_service.cc:140] P fc21155d36d5416fa3650f0268a20b17: Received BeginTabletCopySession request for tablet 70ce4bbee414402aa97c9f3914b9d0c6 from peer 48866680637c40368e45ba844efb9298 ({username='slave'} at 127.31.161.200:55615)
I20240502 01:54:34.473148 7218 tablet_copy_service.cc:161] P fc21155d36d5416fa3650f0268a20b17: Beginning new tablet copy session on tablet 70ce4bbee414402aa97c9f3914b9d0c6 from peer 48866680637c40368e45ba844efb9298 at {username='slave'} at 127.31.161.200:55615: session id = 48866680637c40368e45ba844efb9298-70ce4bbee414402aa97c9f3914b9d0c6
I20240502 01:54:34.476095 7218 tablet_copy_source_session.cc:205] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:34.477486 11338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70ce4bbee414402aa97c9f3914b9d0c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.483142 11338 tablet_copy_client.cc:783] T 70ce4bbee414402aa97c9f3914b9d0c6 P 48866680637c40368e45ba844efb9298: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.483395 11338 tablet_copy_client.cc:647] T 70ce4bbee414402aa97c9f3914b9d0c6 P 48866680637c40368e45ba844efb9298: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.486135 11348 tablet_copy_client.cc:1002] T 70ce4bbee414402aa97c9f3914b9d0c6 P 48866680637c40368e45ba844efb9298: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:34.508154 11365 ts_tablet_manager.cc:902] T db7a48f131f141f0a4e45036434f8129 P 9e47e5c098b848c9abe1559cbcbb6321: Initiating tablet copy from peer 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:34.510304 11365 tablet_copy_client.cc:308] T db7a48f131f141f0a4e45036434f8129 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.194:45315
I20240502 01:54:34.511323 6958 tablet_copy_service.cc:140] P 137e2183ff57411791e9af58dfab0a3c: Received BeginTabletCopySession request for tablet db7a48f131f141f0a4e45036434f8129 from peer 9e47e5c098b848c9abe1559cbcbb6321 ({username='slave'} at 127.31.161.198:33415)
I20240502 01:54:34.511499 6958 tablet_copy_service.cc:161] P 137e2183ff57411791e9af58dfab0a3c: Beginning new tablet copy session on tablet db7a48f131f141f0a4e45036434f8129 from peer 9e47e5c098b848c9abe1559cbcbb6321 at {username='slave'} at 127.31.161.198:33415: session id = 9e47e5c098b848c9abe1559cbcbb6321-db7a48f131f141f0a4e45036434f8129
I20240502 01:54:34.513851 6958 tablet_copy_source_session.cc:205] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:34.515435 11365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db7a48f131f141f0a4e45036434f8129. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.522291 11365 tablet_copy_client.cc:783] T db7a48f131f141f0a4e45036434f8129 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.522554 11365 tablet_copy_client.cc:647] T db7a48f131f141f0a4e45036434f8129 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.525349 11369 tablet_copy_client.cc:1002] T db7a48f131f141f0a4e45036434f8129 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:34.617201 11415 ts_tablet_manager.cc:902] T 5acc7c2405464a328b2016bb2293f7e8 P fdfc60a29ea9465c88ee45c3c332caa1: Initiating tablet copy from peer fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873)
I20240502 01:54:34.636498 11420 ts_tablet_manager.cc:902] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb: Initiating tablet copy from peer 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:34.642213 11424 ts_tablet_manager.cc:902] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1: Initiating tablet copy from peer 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:34.650781 11432 ts_tablet_manager.cc:902] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1: Initiating tablet copy from peer 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:34.652063 11424 tablet_copy_client.cc:308] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.194:45315
I20240502 01:54:34.652652 11422 ts_tablet_manager.cc:902] T d55da66aad2140d788704b056e476e99 P 9e47e5c098b848c9abe1559cbcbb6321: Initiating tablet copy from peer fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873)
I20240502 01:54:34.654664 11415 tablet_copy_client.cc:308] T 5acc7c2405464a328b2016bb2293f7e8 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:38873
I20240502 01:54:34.665712 11432 tablet_copy_client.cc:308] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.194:45315
I20240502 01:54:34.684887 6958 tablet_copy_service.cc:140] P 137e2183ff57411791e9af58dfab0a3c: Received BeginTabletCopySession request for tablet dea582c2912241609405695eed436abe from peer fdfc60a29ea9465c88ee45c3c332caa1 ({username='slave'} at 127.31.161.201:42373)
I20240502 01:54:34.685524 6958 tablet_copy_service.cc:161] P 137e2183ff57411791e9af58dfab0a3c: Beginning new tablet copy session on tablet dea582c2912241609405695eed436abe from peer fdfc60a29ea9465c88ee45c3c332caa1 at {username='slave'} at 127.31.161.201:42373: session id = fdfc60a29ea9465c88ee45c3c332caa1-dea582c2912241609405695eed436abe
I20240502 01:54:34.685595 6957 tablet_copy_service.cc:140] P 137e2183ff57411791e9af58dfab0a3c: Received BeginTabletCopySession request for tablet 8960ca8e5d26429589abc4a154311983 from peer fdfc60a29ea9465c88ee45c3c332caa1 ({username='slave'} at 127.31.161.201:42373)
I20240502 01:54:34.685784 6957 tablet_copy_service.cc:161] P 137e2183ff57411791e9af58dfab0a3c: Beginning new tablet copy session on tablet 8960ca8e5d26429589abc4a154311983 from peer fdfc60a29ea9465c88ee45c3c332caa1 at {username='slave'} at 127.31.161.201:42373: session id = fdfc60a29ea9465c88ee45c3c332caa1-8960ca8e5d26429589abc4a154311983
I20240502 01:54:34.688016 6958 tablet_copy_source_session.cc:205] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:34.689039 11420 tablet_copy_client.cc:308] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:46083
I20240502 01:54:34.689479 6957 tablet_copy_source_session.cc:205] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:34.689936 7089 tablet_copy_service.cc:140] P 11c097d7d32a4c22a6385a7465876eb3: Received BeginTabletCopySession request for tablet 640b6474d1834321881094ed0b11b857 from peer 3f156b7cfb4f40f481699b257153e5cb ({username='slave'} at 127.31.161.197:34771)
I20240502 01:54:34.690109 7089 tablet_copy_service.cc:161] P 11c097d7d32a4c22a6385a7465876eb3: Beginning new tablet copy session on tablet 640b6474d1834321881094ed0b11b857 from peer 3f156b7cfb4f40f481699b257153e5cb at {username='slave'} at 127.31.161.197:34771: session id = 3f156b7cfb4f40f481699b257153e5cb-640b6474d1834321881094ed0b11b857
I20240502 01:54:34.694468 11432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8960ca8e5d26429589abc4a154311983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.697948 7089 tablet_copy_source_session.cc:205] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:34.699012 11424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dea582c2912241609405695eed436abe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.705986 11420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 640b6474d1834321881094ed0b11b857. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.711172 11446 ts_tablet_manager.cc:902] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc: Initiating tablet copy from peer 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:34.712808 11420 tablet_copy_client.cc:783] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.713136 7218 tablet_copy_service.cc:140] P fc21155d36d5416fa3650f0268a20b17: Received BeginTabletCopySession request for tablet 5acc7c2405464a328b2016bb2293f7e8 from peer fdfc60a29ea9465c88ee45c3c332caa1 ({username='slave'} at 127.31.161.201:48813)
I20240502 01:54:34.713320 7218 tablet_copy_service.cc:161] P fc21155d36d5416fa3650f0268a20b17: Beginning new tablet copy session on tablet 5acc7c2405464a328b2016bb2293f7e8 from peer fdfc60a29ea9465c88ee45c3c332caa1 at {username='slave'} at 127.31.161.201:48813: session id = fdfc60a29ea9465c88ee45c3c332caa1-5acc7c2405464a328b2016bb2293f7e8
I20240502 01:54:34.715487 11432 tablet_copy_client.cc:783] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.715790 11432 tablet_copy_client.cc:647] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.716046 7218 tablet_copy_source_session.cc:205] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:34.716538 11420 tablet_copy_client.cc:647] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.717697 11415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5acc7c2405464a328b2016bb2293f7e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.718551 11444 tablet_copy_client.cc:1002] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:34.718729 11424 tablet_copy_client.cc:783] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.718982 11424 tablet_copy_client.cc:647] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.720559 11431 tablet_copy_client.cc:1002] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:34.721474 11445 tablet_copy_client.cc:1002] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:34.724359 11415 tablet_copy_client.cc:783] T 5acc7c2405464a328b2016bb2293f7e8 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.724583 11415 tablet_copy_client.cc:647] T 5acc7c2405464a328b2016bb2293f7e8 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.726200 11433 tablet_copy_client.cc:1002] T 5acc7c2405464a328b2016bb2293f7e8 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:34.734591 11446 tablet_copy_client.cc:308] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:46083
I20240502 01:54:34.735255 11422 tablet_copy_client.cc:308] T d55da66aad2140d788704b056e476e99 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:38873
I20240502 01:54:34.735263 7089 tablet_copy_service.cc:140] P 11c097d7d32a4c22a6385a7465876eb3: Received BeginTabletCopySession request for tablet f6a165cbc26f4defb70866779f83bfdc from peer c39b0639d5fb4d9da09bf1d67d617fcc ({username='slave'} at 127.31.161.199:34537)
I20240502 01:54:34.735435 7089 tablet_copy_service.cc:161] P 11c097d7d32a4c22a6385a7465876eb3: Beginning new tablet copy session on tablet f6a165cbc26f4defb70866779f83bfdc from peer c39b0639d5fb4d9da09bf1d67d617fcc at {username='slave'} at 127.31.161.199:34537: session id = c39b0639d5fb4d9da09bf1d67d617fcc-f6a165cbc26f4defb70866779f83bfdc
I20240502 01:54:34.736469 7218 tablet_copy_service.cc:140] P fc21155d36d5416fa3650f0268a20b17: Received BeginTabletCopySession request for tablet d55da66aad2140d788704b056e476e99 from peer 9e47e5c098b848c9abe1559cbcbb6321 ({username='slave'} at 127.31.161.198:52833)
I20240502 01:54:34.736644 7218 tablet_copy_service.cc:161] P fc21155d36d5416fa3650f0268a20b17: Beginning new tablet copy session on tablet d55da66aad2140d788704b056e476e99 from peer 9e47e5c098b848c9abe1559cbcbb6321 at {username='slave'} at 127.31.161.198:52833: session id = 9e47e5c098b848c9abe1559cbcbb6321-d55da66aad2140d788704b056e476e99
I20240502 01:54:34.738349 7089 tablet_copy_source_session.cc:205] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:34.739531 7218 tablet_copy_source_session.cc:205] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:34.739574 11446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a165cbc26f4defb70866779f83bfdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.740746 11422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d55da66aad2140d788704b056e476e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.745806 11446 tablet_copy_client.cc:783] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.746034 11446 tablet_copy_client.cc:647] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.747733 11458 tablet_copy_client.cc:1002] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:34.748076 11422 tablet_copy_client.cc:783] T d55da66aad2140d788704b056e476e99 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.748445 11422 tablet_copy_client.cc:647] T d55da66aad2140d788704b056e476e99 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.749986 11456 tablet_copy_client.cc:1002] T d55da66aad2140d788704b056e476e99 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:34.781638 11459 ts_tablet_manager.cc:902] T 7d8c4a55e09d41e391b8ba318dfe6896 P c39b0639d5fb4d9da09bf1d67d617fcc: Initiating tablet copy from peer fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873)
I20240502 01:54:34.783262 11459 tablet_copy_client.cc:308] T 7d8c4a55e09d41e391b8ba318dfe6896 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:38873
I20240502 01:54:34.784093 7218 tablet_copy_service.cc:140] P fc21155d36d5416fa3650f0268a20b17: Received BeginTabletCopySession request for tablet 7d8c4a55e09d41e391b8ba318dfe6896 from peer c39b0639d5fb4d9da09bf1d67d617fcc ({username='slave'} at 127.31.161.199:57741)
I20240502 01:54:34.784250 7218 tablet_copy_service.cc:161] P fc21155d36d5416fa3650f0268a20b17: Beginning new tablet copy session on tablet 7d8c4a55e09d41e391b8ba318dfe6896 from peer c39b0639d5fb4d9da09bf1d67d617fcc at {username='slave'} at 127.31.161.199:57741: session id = c39b0639d5fb4d9da09bf1d67d617fcc-7d8c4a55e09d41e391b8ba318dfe6896
I20240502 01:54:34.785243 11465 ts_tablet_manager.cc:902] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321: Initiating tablet copy from peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:34.787358 7218 tablet_copy_source_session.cc:205] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:34.788770 11459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d8c4a55e09d41e391b8ba318dfe6896. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.793525 11465 tablet_copy_client.cc:308] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.193:45489
I20240502 01:54:34.794618 6825 tablet_copy_service.cc:140] P 8214b6c4de6d4aa58a22af551eb8dedd: Received BeginTabletCopySession request for tablet f03b37fd2f844f61a39daadc698c9b29 from peer 9e47e5c098b848c9abe1559cbcbb6321 ({username='slave'} at 127.31.161.198:60753)
I20240502 01:54:34.794770 6825 tablet_copy_service.cc:161] P 8214b6c4de6d4aa58a22af551eb8dedd: Beginning new tablet copy session on tablet f03b37fd2f844f61a39daadc698c9b29 from peer 9e47e5c098b848c9abe1559cbcbb6321 at {username='slave'} at 127.31.161.198:60753: session id = 9e47e5c098b848c9abe1559cbcbb6321-f03b37fd2f844f61a39daadc698c9b29
I20240502 01:54:34.796169 11459 tablet_copy_client.cc:783] T 7d8c4a55e09d41e391b8ba318dfe6896 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.796425 11459 tablet_copy_client.cc:647] T 7d8c4a55e09d41e391b8ba318dfe6896 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.797150 6825 tablet_copy_source_session.cc:205] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:34.798094 11464 tablet_copy_client.cc:1002] T 7d8c4a55e09d41e391b8ba318dfe6896 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:34.798491 11465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f03b37fd2f844f61a39daadc698c9b29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.803287 11465 tablet_copy_client.cc:783] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.803543 11465 tablet_copy_client.cc:647] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.805244 11469 tablet_copy_client.cc:1002] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:34.841470 11470 ts_tablet_manager.cc:902] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321: Initiating tablet copy from peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:34.843091 11470 tablet_copy_client.cc:308] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.193:45489
I20240502 01:54:34.843839 6825 tablet_copy_service.cc:140] P 8214b6c4de6d4aa58a22af551eb8dedd: Received BeginTabletCopySession request for tablet d1f4c94a105741339ba69e641244ac18 from peer 9e47e5c098b848c9abe1559cbcbb6321 ({username='slave'} at 127.31.161.198:60753)
I20240502 01:54:34.844044 6825 tablet_copy_service.cc:161] P 8214b6c4de6d4aa58a22af551eb8dedd: Beginning new tablet copy session on tablet d1f4c94a105741339ba69e641244ac18 from peer 9e47e5c098b848c9abe1559cbcbb6321 at {username='slave'} at 127.31.161.198:60753: session id = 9e47e5c098b848c9abe1559cbcbb6321-d1f4c94a105741339ba69e641244ac18
I20240502 01:54:34.847307 6825 tablet_copy_source_session.cc:205] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:34.848626 11470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f4c94a105741339ba69e641244ac18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.853206 11470 tablet_copy_client.cc:783] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.853471 11470 tablet_copy_client.cc:647] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.854969 11474 tablet_copy_client.cc:1002] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:34.860610 11475 ts_tablet_manager.cc:902] T d58f60c0920e4616850cb246acce1fd7 P 3f156b7cfb4f40f481699b257153e5cb: Initiating tablet copy from peer 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:34.862495 11475 tablet_copy_client.cc:308] T d58f60c0920e4616850cb246acce1fd7 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:46083
I20240502 01:54:34.863250 7089 tablet_copy_service.cc:140] P 11c097d7d32a4c22a6385a7465876eb3: Received BeginTabletCopySession request for tablet d58f60c0920e4616850cb246acce1fd7 from peer 3f156b7cfb4f40f481699b257153e5cb ({username='slave'} at 127.31.161.197:34771)
I20240502 01:54:34.863430 7089 tablet_copy_service.cc:161] P 11c097d7d32a4c22a6385a7465876eb3: Beginning new tablet copy session on tablet d58f60c0920e4616850cb246acce1fd7 from peer 3f156b7cfb4f40f481699b257153e5cb at {username='slave'} at 127.31.161.197:34771: session id = 3f156b7cfb4f40f481699b257153e5cb-d58f60c0920e4616850cb246acce1fd7
I20240502 01:54:34.866432 7089 tablet_copy_source_session.cc:205] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:34.867689 11475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d58f60c0920e4616850cb246acce1fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.873600 11475 tablet_copy_client.cc:783] T d58f60c0920e4616850cb246acce1fd7 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.873827 11475 tablet_copy_client.cc:647] T d58f60c0920e4616850cb246acce1fd7 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.875488 11479 tablet_copy_client.cc:1002] T d58f60c0920e4616850cb246acce1fd7 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:34.901741 11480 ts_tablet_manager.cc:902] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1: Initiating tablet copy from peer 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:34.903298 11480 tablet_copy_client.cc:308] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.194:45315
I20240502 01:54:34.904017 6957 tablet_copy_service.cc:140] P 137e2183ff57411791e9af58dfab0a3c: Received BeginTabletCopySession request for tablet 76fb5e1f69284140a6c6da264a124b3e from peer fdfc60a29ea9465c88ee45c3c332caa1 ({username='slave'} at 127.31.161.201:42373)
I20240502 01:54:34.904188 6957 tablet_copy_service.cc:161] P 137e2183ff57411791e9af58dfab0a3c: Beginning new tablet copy session on tablet 76fb5e1f69284140a6c6da264a124b3e from peer fdfc60a29ea9465c88ee45c3c332caa1 at {username='slave'} at 127.31.161.201:42373: session id = fdfc60a29ea9465c88ee45c3c332caa1-76fb5e1f69284140a6c6da264a124b3e
I20240502 01:54:34.906491 6957 tablet_copy_source_session.cc:205] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:34.907630 11480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76fb5e1f69284140a6c6da264a124b3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.912389 11480 tablet_copy_client.cc:783] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.912760 11480 tablet_copy_client.cc:647] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.914234 11484 tablet_copy_client.cc:1002] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:34.915587 11485 ts_tablet_manager.cc:902] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc: Initiating tablet copy from peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:34.929208 11485 tablet_copy_client.cc:308] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.193:45489
I20240502 01:54:34.930079 6825 tablet_copy_service.cc:140] P 8214b6c4de6d4aa58a22af551eb8dedd: Received BeginTabletCopySession request for tablet 4c1d0974a2fd4a548806b6b6560a89f8 from peer c39b0639d5fb4d9da09bf1d67d617fcc ({username='slave'} at 127.31.161.199:43359)
I20240502 01:54:34.930249 6825 tablet_copy_service.cc:161] P 8214b6c4de6d4aa58a22af551eb8dedd: Beginning new tablet copy session on tablet 4c1d0974a2fd4a548806b6b6560a89f8 from peer c39b0639d5fb4d9da09bf1d67d617fcc at {username='slave'} at 127.31.161.199:43359: session id = c39b0639d5fb4d9da09bf1d67d617fcc-4c1d0974a2fd4a548806b6b6560a89f8
I20240502 01:54:34.933362 6825 tablet_copy_source_session.cc:205] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:34.934718 11485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c1d0974a2fd4a548806b6b6560a89f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.939277 11490 ts_tablet_manager.cc:902] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1: Initiating tablet copy from peer 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315)
I20240502 01:54:34.940832 11490 tablet_copy_client.cc:308] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.194:45315
I20240502 01:54:34.941651 6957 tablet_copy_service.cc:140] P 137e2183ff57411791e9af58dfab0a3c: Received BeginTabletCopySession request for tablet 01e98643312b4a2eb80e3bcb3d95674c from peer fdfc60a29ea9465c88ee45c3c332caa1 ({username='slave'} at 127.31.161.201:42373)
I20240502 01:54:34.941814 6957 tablet_copy_service.cc:161] P 137e2183ff57411791e9af58dfab0a3c: Beginning new tablet copy session on tablet 01e98643312b4a2eb80e3bcb3d95674c from peer fdfc60a29ea9465c88ee45c3c332caa1 at {username='slave'} at 127.31.161.201:42373: session id = fdfc60a29ea9465c88ee45c3c332caa1-01e98643312b4a2eb80e3bcb3d95674c
I20240502 01:54:34.942181 11485 tablet_copy_client.cc:783] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.942425 11485 tablet_copy_client.cc:647] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.944974 6957 tablet_copy_source_session.cc:205] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:34.946421 11489 tablet_copy_client.cc:1002] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:34.946709 11490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e98643312b4a2eb80e3bcb3d95674c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:34.953127 11490 tablet_copy_client.cc:783] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:34.953378 11490 tablet_copy_client.cc:647] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:34.955229 11494 tablet_copy_client.cc:1002] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:35.986541 11338 tablet_copy_client.cc:523] T 70ce4bbee414402aa97c9f3914b9d0c6 P 48866680637c40368e45ba844efb9298: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:35.989244 11338 tablet_bootstrap.cc:492] T 70ce4bbee414402aa97c9f3914b9d0c6 P 48866680637c40368e45ba844efb9298: Bootstrap starting.
I20240502 01:54:36.025888 11365 tablet_copy_client.cc:523] T db7a48f131f141f0a4e45036434f8129 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.028841 11338 tablet_bootstrap.cc:492] T 70ce4bbee414402aa97c9f3914b9d0c6 P 48866680637c40368e45ba844efb9298: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.029292 11338 tablet_bootstrap.cc:492] T 70ce4bbee414402aa97c9f3914b9d0c6 P 48866680637c40368e45ba844efb9298: Bootstrap complete.
I20240502 01:54:36.029582 11338 ts_tablet_manager.cc:1367] T 70ce4bbee414402aa97c9f3914b9d0c6 P 48866680637c40368e45ba844efb9298: Time spent bootstrapping tablet: real 0.040s user 0.035s sys 0.003s
I20240502 01:54:36.029708 11365 tablet_bootstrap.cc:492] T db7a48f131f141f0a4e45036434f8129 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap starting.
I20240502 01:54:36.030417 11338 raft_consensus.cc:348] T 70ce4bbee414402aa97c9f3914b9d0c6 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:36.030711 11338 raft_consensus.cc:729] T 70ce4bbee414402aa97c9f3914b9d0c6 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 48866680637c40368e45ba844efb9298, State: Initialized, Role: LEARNER
I20240502 01:54:36.030918 11338 consensus_queue.cc:260] T 70ce4bbee414402aa97c9f3914b9d0c6 P 48866680637c40368e45ba844efb9298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: true } }
I20240502 01:54:36.038419 11338 ts_tablet_manager.cc:1398] T 70ce4bbee414402aa97c9f3914b9d0c6 P 48866680637c40368e45ba844efb9298: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20240502 01:54:36.039383 7218 tablet_copy_service.cc:342] P fc21155d36d5416fa3650f0268a20b17: Request end of tablet copy session 48866680637c40368e45ba844efb9298-70ce4bbee414402aa97c9f3914b9d0c6 received from {username='slave'} at 127.31.161.200:55615
I20240502 01:54:36.039515 7218 tablet_copy_service.cc:433] P fc21155d36d5416fa3650f0268a20b17: ending tablet copy session 48866680637c40368e45ba844efb9298-70ce4bbee414402aa97c9f3914b9d0c6 on tablet 70ce4bbee414402aa97c9f3914b9d0c6 with peer 48866680637c40368e45ba844efb9298
I20240502 01:54:36.075625 11365 tablet_bootstrap.cc:492] T db7a48f131f141f0a4e45036434f8129 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.076066 11365 tablet_bootstrap.cc:492] T db7a48f131f141f0a4e45036434f8129 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap complete.
I20240502 01:54:36.076334 11365 ts_tablet_manager.cc:1367] T db7a48f131f141f0a4e45036434f8129 P 9e47e5c098b848c9abe1559cbcbb6321: Time spent bootstrapping tablet: real 0.047s user 0.035s sys 0.003s
I20240502 01:54:36.076961 11365 raft_consensus.cc:348] T db7a48f131f141f0a4e45036434f8129 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:36.077212 11365 raft_consensus.cc:729] T db7a48f131f141f0a4e45036434f8129 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e47e5c098b848c9abe1559cbcbb6321, State: Initialized, Role: LEARNER
I20240502 01:54:36.077411 11365 consensus_queue.cc:260] T db7a48f131f141f0a4e45036434f8129 P 9e47e5c098b848c9abe1559cbcbb6321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:36.078256 11365 ts_tablet_manager.cc:1398] T db7a48f131f141f0a4e45036434f8129 P 9e47e5c098b848c9abe1559cbcbb6321: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 01:54:36.079038 6957 tablet_copy_service.cc:342] P 137e2183ff57411791e9af58dfab0a3c: Request end of tablet copy session 9e47e5c098b848c9abe1559cbcbb6321-db7a48f131f141f0a4e45036434f8129 received from {username='slave'} at 127.31.161.198:33415
I20240502 01:54:36.079173 6957 tablet_copy_service.cc:433] P 137e2183ff57411791e9af58dfab0a3c: ending tablet copy session 9e47e5c098b848c9abe1559cbcbb6321-db7a48f131f141f0a4e45036434f8129 on tablet db7a48f131f141f0a4e45036434f8129 with peer 9e47e5c098b848c9abe1559cbcbb6321
I20240502 01:54:36.219110 11432 tablet_copy_client.cc:523] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.220888 11424 tablet_copy_client.cc:523] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.221860 11420 tablet_copy_client.cc:523] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.223922 11432 tablet_bootstrap.cc:492] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap starting.
I20240502 01:54:36.224141 11424 tablet_bootstrap.cc:492] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap starting.
I20240502 01:54:36.224886 11420 tablet_bootstrap.cc:492] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap starting.
I20240502 01:54:36.226553 11415 tablet_copy_client.cc:523] T 5acc7c2405464a328b2016bb2293f7e8 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.230087 11415 tablet_bootstrap.cc:492] T 5acc7c2405464a328b2016bb2293f7e8 P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap starting.
I20240502 01:54:36.240592 11415 log.cc:824] T 5acc7c2405464a328b2016bb2293f7e8 P fdfc60a29ea9465c88ee45c3c332caa1: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:36.248219 11446 tablet_copy_client.cc:523] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.250399 11422 tablet_copy_client.cc:523] T d55da66aad2140d788704b056e476e99 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.252161 11446 tablet_bootstrap.cc:492] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap starting.
I20240502 01:54:36.254614 11422 tablet_bootstrap.cc:492] T d55da66aad2140d788704b056e476e99 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap starting.
I20240502 01:54:36.290772 9085 raft_consensus.cc:1203] T 70ce4bbee414402aa97c9f3914b9d0c6 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:36.291828 11420 tablet_bootstrap.cc:492] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.294524 11420 tablet_bootstrap.cc:492] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap complete.
I20240502 01:54:36.294842 11420 ts_tablet_manager.cc:1367] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb: Time spent bootstrapping tablet: real 0.070s user 0.048s sys 0.012s
I20240502 01:54:36.295840 11420 raft_consensus.cc:348] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:36.296154 11420 raft_consensus.cc:729] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3f156b7cfb4f40f481699b257153e5cb, State: Initialized, Role: LEARNER
I20240502 01:54:36.296350 11420 consensus_queue.cc:260] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:36.298444 11459 tablet_copy_client.cc:523] T 7d8c4a55e09d41e391b8ba318dfe6896 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.304221 11459 tablet_bootstrap.cc:492] T 7d8c4a55e09d41e391b8ba318dfe6896 P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap starting.
I20240502 01:54:36.305608 11465 tablet_copy_client.cc:523] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.310302 11465 tablet_bootstrap.cc:492] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap starting.
I20240502 01:54:36.332947 11420 ts_tablet_manager.cc:1398] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb: Time spent starting tablet: real 0.038s user 0.004s sys 0.028s
I20240502 01:54:36.336354 7089 tablet_copy_service.cc:342] P 11c097d7d32a4c22a6385a7465876eb3: Request end of tablet copy session 3f156b7cfb4f40f481699b257153e5cb-640b6474d1834321881094ed0b11b857 received from {username='slave'} at 127.31.161.197:34771
I20240502 01:54:36.336519 7089 tablet_copy_service.cc:433] P 11c097d7d32a4c22a6385a7465876eb3: ending tablet copy session 3f156b7cfb4f40f481699b257153e5cb-640b6474d1834321881094ed0b11b857 on tablet 640b6474d1834321881094ed0b11b857 with peer 3f156b7cfb4f40f481699b257153e5cb
I20240502 01:54:36.342139 11415 tablet_bootstrap.cc:492] T 5acc7c2405464a328b2016bb2293f7e8 P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.342743 11415 tablet_bootstrap.cc:492] T 5acc7c2405464a328b2016bb2293f7e8 P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap complete.
I20240502 01:54:36.343278 11415 ts_tablet_manager.cc:1367] T 5acc7c2405464a328b2016bb2293f7e8 P fdfc60a29ea9465c88ee45c3c332caa1: Time spent bootstrapping tablet: real 0.113s user 0.051s sys 0.009s
I20240502 01:54:36.344535 11446 tablet_bootstrap.cc:492] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.344931 11446 tablet_bootstrap.cc:492] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap complete.
I20240502 01:54:36.345278 11446 ts_tablet_manager.cc:1367] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc: Time spent bootstrapping tablet: real 0.093s user 0.045s sys 0.007s
I20240502 01:54:36.346025 11446 raft_consensus.cc:348] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:36.346423 11446 raft_consensus.cc:729] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c39b0639d5fb4d9da09bf1d67d617fcc, State: Initialized, Role: LEARNER
I20240502 01:54:36.346618 11446 consensus_queue.cc:260] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:36.354065 11415 raft_consensus.cc:348] T 5acc7c2405464a328b2016bb2293f7e8 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:36.354631 11415 raft_consensus.cc:729] T 5acc7c2405464a328b2016bb2293f7e8 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdfc60a29ea9465c88ee45c3c332caa1, State: Initialized, Role: LEARNER
I20240502 01:54:36.355320 11470 tablet_copy_client.cc:523] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.355254 11415 consensus_queue.cc:260] T 5acc7c2405464a328b2016bb2293f7e8 P fdfc60a29ea9465c88ee45c3c332caa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:36.358836 11470 tablet_bootstrap.cc:492] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap starting.
I20240502 01:54:36.369530 11446 ts_tablet_manager.cc:1398] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc: Time spent starting tablet: real 0.024s user 0.003s sys 0.007s
I20240502 01:54:36.373239 11415 ts_tablet_manager.cc:1398] T 5acc7c2405464a328b2016bb2293f7e8 P fdfc60a29ea9465c88ee45c3c332caa1: Time spent starting tablet: real 0.030s user 0.011s sys 0.001s
I20240502 01:54:36.374634 7218 tablet_copy_service.cc:342] P fc21155d36d5416fa3650f0268a20b17: Request end of tablet copy session fdfc60a29ea9465c88ee45c3c332caa1-5acc7c2405464a328b2016bb2293f7e8 received from {username='slave'} at 127.31.161.201:48813
I20240502 01:54:36.374800 7218 tablet_copy_service.cc:433] P fc21155d36d5416fa3650f0268a20b17: ending tablet copy session fdfc60a29ea9465c88ee45c3c332caa1-5acc7c2405464a328b2016bb2293f7e8 on tablet 5acc7c2405464a328b2016bb2293f7e8 with peer fdfc60a29ea9465c88ee45c3c332caa1
I20240502 01:54:36.375299 7089 tablet_copy_service.cc:342] P 11c097d7d32a4c22a6385a7465876eb3: Request end of tablet copy session c39b0639d5fb4d9da09bf1d67d617fcc-f6a165cbc26f4defb70866779f83bfdc received from {username='slave'} at 127.31.161.199:34537
I20240502 01:54:36.375450 7089 tablet_copy_service.cc:433] P 11c097d7d32a4c22a6385a7465876eb3: ending tablet copy session c39b0639d5fb4d9da09bf1d67d617fcc-f6a165cbc26f4defb70866779f83bfdc on tablet f6a165cbc26f4defb70866779f83bfdc with peer c39b0639d5fb4d9da09bf1d67d617fcc
I20240502 01:54:36.376017 11475 tablet_copy_client.cc:523] T d58f60c0920e4616850cb246acce1fd7 P 3f156b7cfb4f40f481699b257153e5cb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.378026 11424 tablet_bootstrap.cc:492] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.378518 11424 tablet_bootstrap.cc:492] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap complete.
I20240502 01:54:36.378820 11424 ts_tablet_manager.cc:1367] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1: Time spent bootstrapping tablet: real 0.155s user 0.057s sys 0.002s
I20240502 01:54:36.379639 11475 tablet_bootstrap.cc:492] T d58f60c0920e4616850cb246acce1fd7 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap starting.
I20240502 01:54:36.379993 11424 raft_consensus.cc:348] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:36.380381 11424 raft_consensus.cc:729] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdfc60a29ea9465c88ee45c3c332caa1, State: Initialized, Role: LEARNER
I20240502 01:54:36.380592 11424 consensus_queue.cc:260] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:36.381238 11424 ts_tablet_manager.cc:1398] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:54:36.382014 6957 tablet_copy_service.cc:342] P 137e2183ff57411791e9af58dfab0a3c: Request end of tablet copy session fdfc60a29ea9465c88ee45c3c332caa1-dea582c2912241609405695eed436abe received from {username='slave'} at 127.31.161.201:42373
I20240502 01:54:36.382153 6957 tablet_copy_service.cc:433] P 137e2183ff57411791e9af58dfab0a3c: ending tablet copy session fdfc60a29ea9465c88ee45c3c332caa1-dea582c2912241609405695eed436abe on tablet dea582c2912241609405695eed436abe with peer fdfc60a29ea9465c88ee45c3c332caa1
I20240502 01:54:36.385950 11432 tablet_bootstrap.cc:492] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.386366 11432 tablet_bootstrap.cc:492] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap complete.
I20240502 01:54:36.386658 11432 ts_tablet_manager.cc:1367] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1: Time spent bootstrapping tablet: real 0.163s user 0.059s sys 0.004s
I20240502 01:54:36.387444 11432 raft_consensus.cc:348] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:36.387744 11432 raft_consensus.cc:729] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdfc60a29ea9465c88ee45c3c332caa1, State: Initialized, Role: LEARNER
I20240502 01:54:36.387965 11432 consensus_queue.cc:260] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:36.388520 11432 ts_tablet_manager.cc:1398] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:54:36.389227 6957 tablet_copy_service.cc:342] P 137e2183ff57411791e9af58dfab0a3c: Request end of tablet copy session fdfc60a29ea9465c88ee45c3c332caa1-8960ca8e5d26429589abc4a154311983 received from {username='slave'} at 127.31.161.201:42373
I20240502 01:54:36.389369 6957 tablet_copy_service.cc:433] P 137e2183ff57411791e9af58dfab0a3c: ending tablet copy session fdfc60a29ea9465c88ee45c3c332caa1-8960ca8e5d26429589abc4a154311983 on tablet 8960ca8e5d26429589abc4a154311983 with peer fdfc60a29ea9465c88ee45c3c332caa1
I20240502 01:54:36.400300 11422 tablet_bootstrap.cc:492] T d55da66aad2140d788704b056e476e99 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.400617 11465 tablet_bootstrap.cc:492] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.400768 11422 tablet_bootstrap.cc:492] T d55da66aad2140d788704b056e476e99 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap complete.
I20240502 01:54:36.401185 11465 tablet_bootstrap.cc:492] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap complete.
I20240502 01:54:36.401506 11465 ts_tablet_manager.cc:1367] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321: Time spent bootstrapping tablet: real 0.092s user 0.040s sys 0.007s
I20240502 01:54:36.402395 11465 raft_consensus.cc:348] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:36.402784 11465 raft_consensus.cc:729] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e47e5c098b848c9abe1559cbcbb6321, State: Initialized, Role: LEARNER
I20240502 01:54:36.403043 11465 consensus_queue.cc:260] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:36.403692 11465 ts_tablet_manager.cc:1398] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:36.404588 6825 tablet_copy_service.cc:342] P 8214b6c4de6d4aa58a22af551eb8dedd: Request end of tablet copy session 9e47e5c098b848c9abe1559cbcbb6321-f03b37fd2f844f61a39daadc698c9b29 received from {username='slave'} at 127.31.161.198:60753
I20240502 01:54:36.404749 6825 tablet_copy_service.cc:433] P 8214b6c4de6d4aa58a22af551eb8dedd: ending tablet copy session 9e47e5c098b848c9abe1559cbcbb6321-f03b37fd2f844f61a39daadc698c9b29 on tablet f03b37fd2f844f61a39daadc698c9b29 with peer 9e47e5c098b848c9abe1559cbcbb6321
I20240502 01:54:36.401069 11422 ts_tablet_manager.cc:1367] T d55da66aad2140d788704b056e476e99 P 9e47e5c098b848c9abe1559cbcbb6321: Time spent bootstrapping tablet: real 0.147s user 0.048s sys 0.004s
I20240502 01:54:36.413504 11422 raft_consensus.cc:348] T d55da66aad2140d788704b056e476e99 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:36.413782 11422 raft_consensus.cc:729] T d55da66aad2140d788704b056e476e99 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e47e5c098b848c9abe1559cbcbb6321, State: Initialized, Role: LEARNER
I20240502 01:54:36.414013 11422 consensus_queue.cc:260] T d55da66aad2140d788704b056e476e99 P 9e47e5c098b848c9abe1559cbcbb6321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:36.414609 11422 ts_tablet_manager.cc:1398] T d55da66aad2140d788704b056e476e99 P 9e47e5c098b848c9abe1559cbcbb6321: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240502 01:54:36.415375 11480 tablet_copy_client.cc:523] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.419832 11480 tablet_bootstrap.cc:492] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap starting.
I20240502 01:54:36.433285 7218 tablet_copy_service.cc:342] P fc21155d36d5416fa3650f0268a20b17: Request end of tablet copy session 9e47e5c098b848c9abe1559cbcbb6321-d55da66aad2140d788704b056e476e99 received from {username='slave'} at 127.31.161.198:52833
I20240502 01:54:36.433459 7218 tablet_copy_service.cc:433] P fc21155d36d5416fa3650f0268a20b17: ending tablet copy session 9e47e5c098b848c9abe1559cbcbb6321-d55da66aad2140d788704b056e476e99 on tablet d55da66aad2140d788704b056e476e99 with peer 9e47e5c098b848c9abe1559cbcbb6321
I20240502 01:54:36.446803 11485 tablet_copy_client.cc:523] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.449123 11459 tablet_bootstrap.cc:492] T 7d8c4a55e09d41e391b8ba318dfe6896 P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.449630 11459 tablet_bootstrap.cc:492] T 7d8c4a55e09d41e391b8ba318dfe6896 P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap complete.
I20240502 01:54:36.449934 11459 ts_tablet_manager.cc:1367] T 7d8c4a55e09d41e391b8ba318dfe6896 P c39b0639d5fb4d9da09bf1d67d617fcc: Time spent bootstrapping tablet: real 0.146s user 0.046s sys 0.005s
I20240502 01:54:36.450853 11485 tablet_bootstrap.cc:492] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap starting.
I20240502 01:54:36.455607 11490 tablet_copy_client.cc:523] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:36.458519 11470 tablet_bootstrap.cc:492] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.458909 11470 tablet_bootstrap.cc:492] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321: Bootstrap complete.
I20240502 01:54:36.459128 11490 tablet_bootstrap.cc:492] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap starting.
I20240502 01:54:36.459172 11470 ts_tablet_manager.cc:1367] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321: Time spent bootstrapping tablet: real 0.100s user 0.043s sys 0.008s
I20240502 01:54:36.459546 11475 tablet_bootstrap.cc:492] T d58f60c0920e4616850cb246acce1fd7 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.459518 11459 raft_consensus.cc:348] T 7d8c4a55e09d41e391b8ba318dfe6896 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:36.459820 11459 raft_consensus.cc:729] T 7d8c4a55e09d41e391b8ba318dfe6896 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c39b0639d5fb4d9da09bf1d67d617fcc, State: Initialized, Role: LEARNER
I20240502 01:54:36.459961 11475 tablet_bootstrap.cc:492] T d58f60c0920e4616850cb246acce1fd7 P 3f156b7cfb4f40f481699b257153e5cb: Bootstrap complete.
I20240502 01:54:36.459878 11470 raft_consensus.cc:348] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:36.460145 11470 raft_consensus.cc:729] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e47e5c098b848c9abe1559cbcbb6321, State: Initialized, Role: LEARNER
I20240502 01:54:36.460042 11459 consensus_queue.cc:260] T 7d8c4a55e09d41e391b8ba318dfe6896 P c39b0639d5fb4d9da09bf1d67d617fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:36.460235 11475 ts_tablet_manager.cc:1367] T d58f60c0920e4616850cb246acce1fd7 P 3f156b7cfb4f40f481699b257153e5cb: Time spent bootstrapping tablet: real 0.081s user 0.051s sys 0.000s
I20240502 01:54:36.460333 11470 consensus_queue.cc:260] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: true } }
I20240502 01:54:36.460970 11475 raft_consensus.cc:348] T d58f60c0920e4616850cb246acce1fd7 P 3f156b7cfb4f40f481699b257153e5cb [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:36.461248 11475 raft_consensus.cc:729] T d58f60c0920e4616850cb246acce1fd7 P 3f156b7cfb4f40f481699b257153e5cb [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3f156b7cfb4f40f481699b257153e5cb, State: Initialized, Role: LEARNER
I20240502 01:54:36.461447 11475 consensus_queue.cc:260] T d58f60c0920e4616850cb246acce1fd7 P 3f156b7cfb4f40f481699b257153e5cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: true } }
I20240502 01:54:36.462309 11470 ts_tablet_manager.cc:1398] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:36.463025 6825 tablet_copy_service.cc:342] P 8214b6c4de6d4aa58a22af551eb8dedd: Request end of tablet copy session 9e47e5c098b848c9abe1559cbcbb6321-d1f4c94a105741339ba69e641244ac18 received from {username='slave'} at 127.31.161.198:60753
I20240502 01:54:36.463219 6825 tablet_copy_service.cc:433] P 8214b6c4de6d4aa58a22af551eb8dedd: ending tablet copy session 9e47e5c098b848c9abe1559cbcbb6321-d1f4c94a105741339ba69e641244ac18 on tablet d1f4c94a105741339ba69e641244ac18 with peer 9e47e5c098b848c9abe1559cbcbb6321
I20240502 01:54:36.465117 11459 ts_tablet_manager.cc:1398] T 7d8c4a55e09d41e391b8ba318dfe6896 P c39b0639d5fb4d9da09bf1d67d617fcc: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20240502 01:54:36.468382 7218 tablet_copy_service.cc:342] P fc21155d36d5416fa3650f0268a20b17: Request end of tablet copy session c39b0639d5fb4d9da09bf1d67d617fcc-7d8c4a55e09d41e391b8ba318dfe6896 received from {username='slave'} at 127.31.161.199:57741
I20240502 01:54:36.468458 11475 ts_tablet_manager.cc:1398] T d58f60c0920e4616850cb246acce1fd7 P 3f156b7cfb4f40f481699b257153e5cb: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240502 01:54:36.468536 7218 tablet_copy_service.cc:433] P fc21155d36d5416fa3650f0268a20b17: ending tablet copy session c39b0639d5fb4d9da09bf1d67d617fcc-7d8c4a55e09d41e391b8ba318dfe6896 on tablet 7d8c4a55e09d41e391b8ba318dfe6896 with peer c39b0639d5fb4d9da09bf1d67d617fcc
I20240502 01:54:36.470443 7089 tablet_copy_service.cc:342] P 11c097d7d32a4c22a6385a7465876eb3: Request end of tablet copy session 3f156b7cfb4f40f481699b257153e5cb-d58f60c0920e4616850cb246acce1fd7 received from {username='slave'} at 127.31.161.197:34771
I20240502 01:54:36.470592 7089 tablet_copy_service.cc:433] P 11c097d7d32a4c22a6385a7465876eb3: ending tablet copy session 3f156b7cfb4f40f481699b257153e5cb-d58f60c0920e4616850cb246acce1fd7 on tablet d58f60c0920e4616850cb246acce1fd7 with peer 3f156b7cfb4f40f481699b257153e5cb
I20240502 01:54:36.502199 11480 tablet_bootstrap.cc:492] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.502710 11480 tablet_bootstrap.cc:492] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap complete.
I20240502 01:54:36.503012 11480 ts_tablet_manager.cc:1367] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1: Time spent bootstrapping tablet: real 0.083s user 0.048s sys 0.000s
I20240502 01:54:36.504364 11480 raft_consensus.cc:348] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:36.504665 11480 raft_consensus.cc:729] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdfc60a29ea9465c88ee45c3c332caa1, State: Initialized, Role: LEARNER
I20240502 01:54:36.505414 11480 consensus_queue.cc:260] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:36.506521 11480 ts_tablet_manager.cc:1398] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 01:54:36.507193 6957 tablet_copy_service.cc:342] P 137e2183ff57411791e9af58dfab0a3c: Request end of tablet copy session fdfc60a29ea9465c88ee45c3c332caa1-76fb5e1f69284140a6c6da264a124b3e received from {username='slave'} at 127.31.161.201:42373
I20240502 01:54:36.507334 6957 tablet_copy_service.cc:433] P 137e2183ff57411791e9af58dfab0a3c: ending tablet copy session fdfc60a29ea9465c88ee45c3c332caa1-76fb5e1f69284140a6c6da264a124b3e on tablet 76fb5e1f69284140a6c6da264a124b3e with peer fdfc60a29ea9465c88ee45c3c332caa1
I20240502 01:54:36.529873 11485 tablet_bootstrap.cc:492] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.530437 11485 tablet_bootstrap.cc:492] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc: Bootstrap complete.
I20240502 01:54:36.530758 11485 ts_tablet_manager.cc:1367] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc: Time spent bootstrapping tablet: real 0.080s user 0.038s sys 0.012s
I20240502 01:54:36.530949 11490 tablet_bootstrap.cc:492] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:36.531368 11490 tablet_bootstrap.cc:492] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap complete.
I20240502 01:54:36.531644 11490 ts_tablet_manager.cc:1367] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1: Time spent bootstrapping tablet: real 0.073s user 0.040s sys 0.008s
I20240502 01:54:36.531543 11485 raft_consensus.cc:348] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:36.531813 11485 raft_consensus.cc:729] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c39b0639d5fb4d9da09bf1d67d617fcc, State: Initialized, Role: LEARNER
I20240502 01:54:36.531999 11485 consensus_queue.cc:260] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: NON_VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: true } }
I20240502 01:54:36.532635 11485 ts_tablet_manager.cc:1398] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:54:36.532771 11490 raft_consensus.cc:348] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:36.533057 11490 raft_consensus.cc:729] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdfc60a29ea9465c88ee45c3c332caa1, State: Initialized, Role: LEARNER
I20240502 01:54:36.533353 11490 consensus_queue.cc:260] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:36.533885 6825 tablet_copy_service.cc:342] P 8214b6c4de6d4aa58a22af551eb8dedd: Request end of tablet copy session c39b0639d5fb4d9da09bf1d67d617fcc-4c1d0974a2fd4a548806b6b6560a89f8 received from {username='slave'} at 127.31.161.199:43359
I20240502 01:54:36.534071 6825 tablet_copy_service.cc:433] P 8214b6c4de6d4aa58a22af551eb8dedd: ending tablet copy session c39b0639d5fb4d9da09bf1d67d617fcc-4c1d0974a2fd4a548806b6b6560a89f8 on tablet 4c1d0974a2fd4a548806b6b6560a89f8 with peer c39b0639d5fb4d9da09bf1d67d617fcc
I20240502 01:54:36.534287 11490 ts_tablet_manager.cc:1398] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:54:36.534658 8957 raft_consensus.cc:1203] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:36.534976 6957 tablet_copy_service.cc:342] P 137e2183ff57411791e9af58dfab0a3c: Request end of tablet copy session fdfc60a29ea9465c88ee45c3c332caa1-01e98643312b4a2eb80e3bcb3d95674c received from {username='slave'} at 127.31.161.201:42373
I20240502 01:54:36.535116 6957 tablet_copy_service.cc:433] P 137e2183ff57411791e9af58dfab0a3c: ending tablet copy session fdfc60a29ea9465c88ee45c3c332caa1-01e98643312b4a2eb80e3bcb3d95674c on tablet 01e98643312b4a2eb80e3bcb3d95674c with peer fdfc60a29ea9465c88ee45c3c332caa1
I20240502 01:54:36.571290 10009 raft_consensus.cc:1203] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:36.603026 8957 raft_consensus.cc:1203] T 7d8c4a55e09d41e391b8ba318dfe6896 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:36.621958 8957 raft_consensus.cc:1203] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:36.638259 8824 raft_consensus.cc:1203] T db7a48f131f141f0a4e45036434f8129 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:36.648131 8691 raft_consensus.cc:1203] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:36.668746 10009 raft_consensus.cc:1203] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:36.679117 10009 raft_consensus.cc:1203] T 5acc7c2405464a328b2016bb2293f7e8 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:36.696810 11506 raft_consensus.cc:1050] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17: attempting to promote NON_VOTER 48866680637c40368e45ba844efb9298 to VOTER
I20240502 01:54:36.697557 11506 consensus_queue.cc:237] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:36.699152 8824 raft_consensus.cc:1203] T d55da66aad2140d788704b056e476e99 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:36.706264 9085 raft_consensus.cc:1261] T 70ce4bbee414402aa97c9f3914b9d0c6 P 48866680637c40368e45ba844efb9298 [term 1 LEARNER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:36.706434 8691 raft_consensus.cc:1261] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:36.706439 6935 raft_consensus.cc:1261] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:36.707007 11506 consensus_queue.cc:1035] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:36.707378 11280 consensus_queue.cc:1035] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:36.708050 11903 consensus_queue.cc:1035] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:36.709357 10009 raft_consensus.cc:1203] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:36.713450 11901 raft_consensus.cc:2932] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:36.714073 8691 raft_consensus.cc:2932] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:36.714123 9085 raft_consensus.cc:2932] T 70ce4bbee414402aa97c9f3914b9d0c6 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:36.719364 6654 catalog_manager.cc:5484] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 7 to 8, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:36.719492 6935 raft_consensus.cc:2932] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 48866680637c40368e45ba844efb9298 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:36.742076 8824 raft_consensus.cc:1203] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:36.867192 8691 raft_consensus.cc:1203] T d58f60c0920e4616850cb246acce1fd7 P 3f156b7cfb4f40f481699b257153e5cb [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240502 01:54:36.963744 8824 raft_consensus.cc:1203] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:36.982151 10009 raft_consensus.cc:1203] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:37.016110 7197 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "70ce4bbee414402aa97c9f3914b9d0c6"
dest_uuid: "fc21155d36d5416fa3650f0268a20b17"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40316
I20240502 01:54:37.016294 7197 raft_consensus.cc:595] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:37.026695 11506 raft_consensus.cc:1050] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17: attempting to promote NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc to VOTER
I20240502 01:54:37.027590 11506 consensus_queue.cc:237] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:37.030221 11993 raft_consensus.cc:1050] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd: attempting to promote NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc to VOTER
I20240502 01:54:37.030531 6801 raft_consensus.cc:1261] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.030861 9085 raft_consensus.cc:1261] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.031244 8957 raft_consensus.cc:1261] T 7d8c4a55e09d41e391b8ba318dfe6896 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.031111 11993 consensus_queue.cc:237] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:37.031437 11280 consensus_queue.cc:1035] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.031823 11901 consensus_queue.cc:1035] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.032269 11903 consensus_queue.cc:1035] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.036659 11903 raft_consensus.cc:2932] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:37.037712 8957 raft_consensus.cc:1261] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.038120 8956 raft_consensus.cc:2932] T 7d8c4a55e09d41e391b8ba318dfe6896 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:37.041467 6935 raft_consensus.cc:1261] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.043063 6801 raft_consensus.cc:2932] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:37.043511 6654 catalog_manager.cc:5484] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 7 to 8, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:37.045799 9086 raft_consensus.cc:1261] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.047441 11993 consensus_queue.cc:1035] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.048063 11627 consensus_queue.cc:1035] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.048202 9086 raft_consensus.cc:2932] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:37.053027 7197 consensus_queue.cc:237] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:37.055148 9086 raft_consensus.cc:1261] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:37.055642 8956 raft_consensus.cc:1261] T 7d8c4a55e09d41e391b8ba318dfe6896 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:37.055665 11903 consensus_queue.cc:1035] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.056216 11903 consensus_queue.cc:1035] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.056320 11993 consensus_queue.cc:1035] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.058355 11901 raft_consensus.cc:2932] T 7d8c4a55e09d41e391b8ba318dfe6896 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:37.058768 9086 raft_consensus.cc:2932] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:37.061033 11627 raft_consensus.cc:2932] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:37.061934 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d8c4a55e09d41e391b8ba318dfe6896 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:37.064419 6657 catalog_manager.cc:5484] T 7d8c4a55e09d41e391b8ba318dfe6896 P 48866680637c40368e45ba844efb9298 reported cstate change: config changed from index 8 to 9, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:37.066498 8956 raft_consensus.cc:2932] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:37.067242 6653 catalog_manager.cc:5484] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index 7 to 8, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:37.068828 6935 raft_consensus.cc:2932] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:37.070034 9085 raft_consensus.cc:2932] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:37.070936 6782 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d8c4a55e09d41e391b8ba318dfe6896 with delete_type TABLET_DATA_TOMBSTONED (TS 8214b6c4de6d4aa58a22af551eb8dedd not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35388
I20240502 01:54:37.074455 12003 tablet_replica.cc:330] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd: stopping tablet replica
I20240502 01:54:37.074695 12003 raft_consensus.cc:2229] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:37.075016 12003 raft_consensus.cc:2258] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:37.076524 12003 ts_tablet_manager.cc:1875] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:37.078387 8956 raft_consensus.cc:2932] T 7d8c4a55e09d41e391b8ba318dfe6896 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:37.082816 12003 ts_tablet_manager.cc:1888] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:37.082924 12003 log.cc:1196] T 7d8c4a55e09d41e391b8ba318dfe6896 P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/7d8c4a55e09d41e391b8ba318dfe6896
I20240502 01:54:37.083578 6645 catalog_manager.cc:4830] TS 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): tablet 7d8c4a55e09d41e391b8ba318dfe6896 (table rebalance_test_table_8 [id=8abdf44bc9e2414b9e4f80bf4c87c953]) successfully deleted
I20240502 01:54:37.089112 11901 raft_consensus.cc:1050] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17: attempting to promote NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 to VOTER
I20240502 01:54:37.090023 11901 consensus_queue.cc:237] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:37.092247 8824 raft_consensus.cc:1261] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.092278 6935 raft_consensus.cc:1261] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.092365 10009 raft_consensus.cc:1261] T 5acc7c2405464a328b2016bb2293f7e8 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.092783 11280 consensus_queue.cc:1035] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.093118 11903 consensus_queue.cc:1035] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.093369 11901 consensus_queue.cc:1035] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.096536 11280 raft_consensus.cc:2932] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.097164 6935 raft_consensus.cc:2932] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.098292 10009 raft_consensus.cc:2932] T 5acc7c2405464a328b2016bb2293f7e8 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.100314 8824 raft_consensus.cc:2932] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.101837 6654 catalog_manager.cc:5484] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:37.107239 7197 consensus_queue.cc:237] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:37.109517 10009 raft_consensus.cc:1261] T 5acc7c2405464a328b2016bb2293f7e8 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:37.109717 8824 raft_consensus.cc:1261] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:37.110311 11966 raft_consensus.cc:1050] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c: attempting to promote NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 to VOTER
I20240502 01:54:37.111145 11966 consensus_queue.cc:237] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:37.111408 11903 consensus_queue.cc:1035] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.111922 11903 consensus_queue.cc:1035] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.114120 7197 raft_consensus.cc:1261] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.114125 10008 raft_consensus.cc:1261] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.114001 11280 raft_consensus.cc:2932] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.114708 9085 raft_consensus.cc:1261] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.114574 8824 raft_consensus.cc:2932] T 5acc7c2405464a328b2016bb2293f7e8 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.114817 11966 consensus_queue.cc:1035] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.115523 11500 consensus_queue.cc:1035] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.116107 11500 consensus_queue.cc:1035] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.118106 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5acc7c2405464a328b2016bb2293f7e8 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:37.120062 6659 catalog_manager.cc:5484] T 5acc7c2405464a328b2016bb2293f7e8 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 8 to 9, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:37.123920 10009 raft_consensus.cc:2932] T 5acc7c2405464a328b2016bb2293f7e8 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.125516 6915 tablet_service.cc:1514] Processing DeleteTablet for tablet 5acc7c2405464a328b2016bb2293f7e8 with delete_type TABLET_DATA_TOMBSTONED (TS 137e2183ff57411791e9af58dfab0a3c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:58498
I20240502 01:54:37.125725 11500 raft_consensus.cc:1050] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c: attempting to promote NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 to VOTER
I20240502 01:54:37.126497 11500 consensus_queue.cc:237] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:37.129037 11627 raft_consensus.cc:1050] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd: attempting to promote NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 to VOTER
I20240502 01:54:37.129969 11627 consensus_queue.cc:237] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:37.132349 8823 raft_consensus.cc:1261] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.132956 11993 consensus_queue.cc:1035] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.133258 6935 raft_consensus.cc:1261] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.134120 11627 consensus_queue.cc:1035] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.138862 11903 raft_consensus.cc:979] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17: : Instructing follower 3f156b7cfb4f40f481699b257153e5cb to start an election
I20240502 01:54:37.139024 11901 raft_consensus.cc:1067] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Signalling peer 3f156b7cfb4f40f481699b257153e5cb to start an election
I20240502 01:54:37.139787 8691 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "70ce4bbee414402aa97c9f3914b9d0c6"
dest_uuid: "3f156b7cfb4f40f481699b257153e5cb"
from {username='slave'} at 127.31.161.196:37635
I20240502 01:54:37.140136 8691 raft_consensus.cc:482] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:37.140321 8691 raft_consensus.cc:3037] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:37.143451 8691 raft_consensus.cc:504] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:37.144485 11993 raft_consensus.cc:2932] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.146207 6935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ce4bbee414402aa97c9f3914b9d0c6" candidate_uuid: "3f156b7cfb4f40f481699b257153e5cb" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "137e2183ff57411791e9af58dfab0a3c"
I20240502 01:54:37.146447 6935 raft_consensus.cc:3037] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:37.148967 6936 raft_consensus.cc:2932] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.149637 8824 raft_consensus.cc:2932] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.150753 6659 catalog_manager.cc:5484] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index 7 to 8, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:37.154054 6935 raft_consensus.cc:2454] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f156b7cfb4f40f481699b257153e5cb in term 2.
I20240502 01:54:37.157127 12013 tablet_replica.cc:330] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c: stopping tablet replica
I20240502 01:54:37.157347 12013 raft_consensus.cc:2229] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:37.157765 12013 raft_consensus.cc:2258] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:37.161271 7197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ce4bbee414402aa97c9f3914b9d0c6" candidate_uuid: "3f156b7cfb4f40f481699b257153e5cb" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "fc21155d36d5416fa3650f0268a20b17"
I20240502 01:54:37.160805 10008 raft_consensus.cc:2932] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.161509 7197 raft_consensus.cc:3032] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:37.161594 7197 raft_consensus.cc:729] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Running, Role: LEADER
I20240502 01:54:37.161917 7197 consensus_queue.cc:260] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:37.162335 7197 raft_consensus.cc:3037] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:37.164918 7197 raft_consensus.cc:2454] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f156b7cfb4f40f481699b257153e5cb in term 2.
I20240502 01:54:37.164404 12008 raft_consensus.cc:2932] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.169728 12013 ts_tablet_manager.cc:1875] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:37.171669 6657 catalog_manager.cc:5484] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1 reported cstate change: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.174183 11966 raft_consensus.cc:1050] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c: attempting to promote NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 to VOTER
I20240502 01:54:37.175436 8625 leader_election.cc:302] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 137e2183ff57411791e9af58dfab0a3c, 3f156b7cfb4f40f481699b257153e5cb, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:37.176023 11815 raft_consensus.cc:2781] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:37.175523 7197 raft_consensus.cc:2932] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.175006 11966 consensus_queue.cc:237] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:37.182130 8691 leader_election.cc:288] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [CANDIDATE]: Term 2 election: Requested vote from peers 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315), fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873), 48866680637c40368e45ba844efb9298 (127.31.161.200:34863)
I20240502 01:54:37.185003 8956 raft_consensus.cc:1261] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.186666 11993 consensus_queue.cc:1035] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.187878 12013 ts_tablet_manager.cc:1888] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:37.188930 12013 log.cc:1196] T 5acc7c2405464a328b2016bb2293f7e8 P 137e2183ff57411791e9af58dfab0a3c: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/wals/5acc7c2405464a328b2016bb2293f7e8
I20240502 01:54:37.190068 6646 catalog_manager.cc:4830] TS 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315): tablet 5acc7c2405464a328b2016bb2293f7e8 (table rebalance_test_table_1 [id=9a08c31578ae4224915289c7c289ee7a]) successfully deleted
I20240502 01:54:37.201812 11280 raft_consensus.cc:1050] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17: attempting to promote NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 to VOTER
I20240502 01:54:37.202634 11280 consensus_queue.cc:237] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:37.212296 11815 raft_consensus.cc:686] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [term 2 LEADER]: Becoming Leader. State: Replica: 3f156b7cfb4f40f481699b257153e5cb, State: Running, Role: LEADER
I20240502 01:54:37.212826 10008 raft_consensus.cc:1261] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.212908 11815 consensus_queue.cc:237] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:37.212765 8956 raft_consensus.cc:2932] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.213765 12016 consensus_queue.cc:1035] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.215111 8824 raft_consensus.cc:1261] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.215313 7197 raft_consensus.cc:1261] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.219667 12016 consensus_queue.cc:1035] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.220474 12025 consensus_queue.cc:1035] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.225785 12021 raft_consensus.cc:1050] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3: attempting to promote NON_VOTER 3f156b7cfb4f40f481699b257153e5cb to VOTER
I20240502 01:54:37.226791 12021 consensus_queue.cc:237] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:37.231842 11564 raft_consensus.cc:1050] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3: attempting to promote NON_VOTER c39b0639d5fb4d9da09bf1d67d617fcc to VOTER
I20240502 01:54:37.232872 8823 raft_consensus.cc:1261] T d55da66aad2140d788704b056e476e99 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.233649 11901 consensus_queue.cc:1035] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.239185 11564 consensus_queue.cc:237] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:37.248009 8691 raft_consensus.cc:1261] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb [term 1 LEARNER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.249318 11564 consensus_queue.cc:1035] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.249450 7066 raft_consensus.cc:1261] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.250749 11280 consensus_queue.cc:1035] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.252265 6801 raft_consensus.cc:1261] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.250592 8824 raft_consensus.cc:2932] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.254794 8956 raft_consensus.cc:1261] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 LEARNER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.255964 8691 raft_consensus.cc:1261] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.255954 12021 consensus_queue.cc:1035] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.256366 12027 consensus_queue.cc:1035] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.256470 11901 consensus_queue.cc:1035] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.258353 9087 raft_consensus.cc:1261] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.258983 9086 raft_consensus.cc:1261] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.262373 8822 raft_consensus.cc:1261] T db7a48f131f141f0a4e45036434f8129 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.264878 6657 catalog_manager.cc:5484] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb reported cstate change: term changed from 1 to 2, leader changed from fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) to 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197). New cstate: current_term: 2 leader_uuid: "3f156b7cfb4f40f481699b257153e5cb" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:37.273444 12051 consensus_queue.cc:1035] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.274312 8692 raft_consensus.cc:1261] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.274835 12016 consensus_queue.cc:1035] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 01:54:37.275105 6870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:45489, user_credentials={real_user=slave}} blocked reactor thread for 46303.8us
I20240502 01:54:37.277583 12050 raft_consensus.cc:1050] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c: attempting to promote NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 to VOTER
I20240502 01:54:37.278475 12050 consensus_queue.cc:237] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:37.278502 9086 raft_consensus.cc:2932] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.260906 12043 raft_consensus.cc:2932] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.283627 12021 raft_consensus.cc:2932] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:37.288372 11901 raft_consensus.cc:2932] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.289222 8957 raft_consensus.cc:2932] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:37.290045 10008 raft_consensus.cc:1261] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.290944 10008 raft_consensus.cc:2932] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.292173 6657 catalog_manager.cc:5484] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321 reported cstate change: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.294575 6653 catalog_manager.cc:5484] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc reported cstate change: config changed from index 7 to 8, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:37.296085 6659 catalog_manager.cc:5484] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 7 to 8, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:37.292748 8824 raft_consensus.cc:2932] T d55da66aad2140d788704b056e476e99 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.300904 12037 consensus_queue.cc:1035] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.303753 9086 raft_consensus.cc:1261] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.304353 12051 consensus_queue.cc:1035] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.305796 6801 raft_consensus.cc:1261] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.306071 6803 raft_consensus.cc:2932] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:37.306636 6801 raft_consensus.cc:1261] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.307608 12050 consensus_queue.cc:1035] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.308049 12050 consensus_queue.cc:1035] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.309674 7066 raft_consensus.cc:2932] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.311055 7197 raft_consensus.cc:1261] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.312562 12041 consensus_queue.cc:1035] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.314935 7198 raft_consensus.cc:2932] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.319401 12051 raft_consensus.cc:2932] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.319957 10009 raft_consensus.cc:2932] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.324820 7198 consensus_queue.cc:237] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:37.325825 6653 catalog_manager.cc:5484] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 reported cstate change: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.327437 8692 raft_consensus.cc:1261] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:37.328155 11901 consensus_queue.cc:1035] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.328332 8692 raft_consensus.cc:2932] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.332201 8823 raft_consensus.cc:1261] T d55da66aad2140d788704b056e476e99 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:37.332943 11280 consensus_queue.cc:1035] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.334151 11903 raft_consensus.cc:2932] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.335844 6803 raft_consensus.cc:2932] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.338138 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d55da66aad2140d788704b056e476e99 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:37.339291 8692 raft_consensus.cc:2932] T d55da66aad2140d788704b056e476e99 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.340853 6653 catalog_manager.cc:5484] T d55da66aad2140d788704b056e476e99 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 8 to 9, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New cstate: current_term: 1 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:37.342276 8821 raft_consensus.cc:2932] T d55da66aad2140d788704b056e476e99 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.352154 7047 tablet_service.cc:1514] Processing DeleteTablet for tablet d55da66aad2140d788704b056e476e99 with delete_type TABLET_DATA_TOMBSTONED (TS 11c097d7d32a4c22a6385a7465876eb3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:59762
I20240502 01:54:37.352630 9086 raft_consensus.cc:2932] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.355592 6935 consensus_queue.cc:237] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:37.356365 12020 consensus_queue.cc:1035] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.358755 10008 raft_consensus.cc:1261] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:37.358839 8823 raft_consensus.cc:1261] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:37.359537 11500 consensus_queue.cc:1035] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.360054 12037 consensus_queue.cc:1035] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.363016 11627 raft_consensus.cc:1050] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd: attempting to promote NON_VOTER 9e47e5c098b848c9abe1559cbcbb6321 to VOTER
I20240502 01:54:37.363835 11627 consensus_queue.cc:237] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:37.364441 12037 raft_consensus.cc:2932] T 01e98643312b4a2eb80e3bcb3d95674c P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.356366 12041 consensus_queue.cc:1035] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.013s
I20240502 01:54:37.370450 10008 raft_consensus.cc:2932] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.371016 8820 raft_consensus.cc:2932] T 01e98643312b4a2eb80e3bcb3d95674c P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.370543 12028 raft_consensus.cc:2932] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:37.376096 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 01e98643312b4a2eb80e3bcb3d95674c with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:37.375702 7197 raft_consensus.cc:2932] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:37.377501 12050 raft_consensus.cc:2932] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.378587 6658 catalog_manager.cc:5484] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1 reported cstate change: config changed from index 8 to 9, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.379259 8692 raft_consensus.cc:2932] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:37.382365 8691 raft_consensus.cc:2932] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.383612 6801 raft_consensus.cc:2932] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.384873 6935 consensus_queue.cc:237] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:37.387324 10008 raft_consensus.cc:1261] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:37.388792 7179 tablet_service.cc:1514] Processing DeleteTablet for tablet 01e98643312b4a2eb80e3bcb3d95674c with delete_type TABLET_DATA_TOMBSTONED (TS fc21155d36d5416fa3650f0268a20b17 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:40840
I20240502 01:54:37.389899 6654 catalog_manager.cc:5484] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb reported cstate change: config changed from index 7 to 8, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:37.390240 9086 raft_consensus.cc:1261] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:37.388556 12037 consensus_queue.cc:1035] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.005s
I20240502 01:54:37.394227 12066 tablet_replica.cc:330] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17: stopping tablet replica
I20240502 01:54:37.394167 11966 consensus_queue.cc:1035] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.396761 6654 catalog_manager.cc:5484] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb reported cstate change: config changed from index 7 to 8, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.396674 12049 raft_consensus.cc:2932] T dea582c2912241609405695eed436abe P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.400672 12066 raft_consensus.cc:2229] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:37.401103 12066 raft_consensus.cc:2258] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:37.402469 12066 ts_tablet_manager.cc:1875] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:37.403352 9087 raft_consensus.cc:2932] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.407099 10008 raft_consensus.cc:2932] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.409598 6653 catalog_manager.cc:5484] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298 reported cstate change: config changed from index 8 to 9, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.411293 8823 raft_consensus.cc:2932] T db7a48f131f141f0a4e45036434f8129 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.413530 7179 tablet_service.cc:1514] Processing DeleteTablet for tablet dea582c2912241609405695eed436abe with delete_type TABLET_DATA_TOMBSTONED (TS fc21155d36d5416fa3650f0268a20b17 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:40840
I20240502 01:54:37.418081 12066 ts_tablet_manager.cc:1888] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:37.418642 12066 log.cc:1196] T 01e98643312b4a2eb80e3bcb3d95674c P fc21155d36d5416fa3650f0268a20b17: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/01e98643312b4a2eb80e3bcb3d95674c
I20240502 01:54:37.418908 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dea582c2912241609405695eed436abe with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:37.419986 12066 tablet_replica.cc:330] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17: stopping tablet replica
I20240502 01:54:37.420020 6645 catalog_manager.cc:4830] TS fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873): tablet 01e98643312b4a2eb80e3bcb3d95674c (table rebalance_test_table_3 [id=76b4b12b25de439c840462959d467432]) successfully deleted
I20240502 01:54:37.422972 12066 raft_consensus.cc:2229] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:37.423522 12066 raft_consensus.cc:2258] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:37.426455 12066 ts_tablet_manager.cc:1875] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:37.431284 9087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ce4bbee414402aa97c9f3914b9d0c6" candidate_uuid: "3f156b7cfb4f40f481699b257153e5cb" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "48866680637c40368e45ba844efb9298"
I20240502 01:54:37.431631 9087 raft_consensus.cc:3037] T 70ce4bbee414402aa97c9f3914b9d0c6 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:37.434248 9085 raft_consensus.cc:2932] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:37.430920 6935 consensus_queue.cc:237] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:37.435273 12066 ts_tablet_manager.cc:1888] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:37.434644 9087 raft_consensus.cc:2454] T 70ce4bbee414402aa97c9f3914b9d0c6 P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f156b7cfb4f40f481699b257153e5cb in term 2.
I20240502 01:54:37.437911 8691 raft_consensus.cc:1261] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:37.438375 8823 raft_consensus.cc:1261] T db7a48f131f141f0a4e45036434f8129 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:37.438577 12037 consensus_queue.cc:1035] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.438948 8956 raft_consensus.cc:1261] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.439018 8824 raft_consensus.cc:1261] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 LEARNER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.439433 6936 raft_consensus.cc:1261] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.439524 11993 consensus_queue.cc:1035] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.440026 11992 consensus_queue.cc:1035] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.440692 12051 consensus_queue.cc:1035] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.440721 11999 consensus_queue.cc:1035] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.441097 12037 raft_consensus.cc:2932] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.442077 9086 raft_consensus.cc:2932] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:37.445315 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet db7a48f131f141f0a4e45036434f8129 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:37.446099 8691 raft_consensus.cc:2932] T db7a48f131f141f0a4e45036434f8129 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.448288 6653 catalog_manager.cc:5484] T db7a48f131f141f0a4e45036434f8129 P 137e2183ff57411791e9af58dfab0a3c reported cstate change: config changed from index 8 to 9, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:37.452692 12062 tablet_replica.cc:330] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3: stopping tablet replica
I20240502 01:54:37.452946 12062 raft_consensus.cc:2229] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:37.453575 12066 log.cc:1196] T dea582c2912241609405695eed436abe P fc21155d36d5416fa3650f0268a20b17: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/dea582c2912241609405695eed436abe
I20240502 01:54:37.454232 12062 raft_consensus.cc:2258] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:37.456290 12062 ts_tablet_manager.cc:1875] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:37.457203 6782 tablet_service.cc:1514] Processing DeleteTablet for tablet db7a48f131f141f0a4e45036434f8129 with delete_type TABLET_DATA_TOMBSTONED (TS 8214b6c4de6d4aa58a22af551eb8dedd not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35388
I20240502 01:54:37.457566 12003 tablet_replica.cc:330] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd: stopping tablet replica
I20240502 01:54:37.457791 12003 raft_consensus.cc:2229] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:37.458034 12003 raft_consensus.cc:2258] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:37.459297 12003 ts_tablet_manager.cc:1875] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:37.458382 8824 raft_consensus.cc:2932] T db7a48f131f141f0a4e45036434f8129 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.467288 12028 raft_consensus.cc:1050] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3: attempting to promote NON_VOTER 3f156b7cfb4f40f481699b257153e5cb to VOTER
I20240502 01:54:37.468180 12003 ts_tablet_manager.cc:1888] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:37.468324 12028 consensus_queue.cc:237] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:37.472190 8691 raft_consensus.cc:1261] T d58f60c0920e4616850cb246acce1fd7 P 3f156b7cfb4f40f481699b257153e5cb [term 2 LEARNER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:37.472427 12003 log.cc:1196] T db7a48f131f141f0a4e45036434f8129 P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/db7a48f131f141f0a4e45036434f8129
I20240502 01:54:37.472592 6801 raft_consensus.cc:1261] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 2 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:37.472898 12028 consensus_queue.cc:1035] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.473268 12020 consensus_queue.cc:1035] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.473752 6645 catalog_manager.cc:4830] TS 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): tablet db7a48f131f141f0a4e45036434f8129 (table rebalance_test_table_6 [id=981b993f808c4320845154760fb7959f]) successfully deleted
I20240502 01:54:37.477262 12062 ts_tablet_manager.cc:1888] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:37.479041 6645 catalog_manager.cc:4830] TS fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873): tablet dea582c2912241609405695eed436abe (table rebalance_test_table_2 [id=3316a56a8d6043c6b05160d6e55a33e0]) successfully deleted
I20240502 01:54:37.479629 8823 raft_consensus.cc:1261] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [term 2 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:37.480171 12028 consensus_queue.cc:1035] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.481918 12062 log.cc:1196] T d55da66aad2140d788704b056e476e99 P 11c097d7d32a4c22a6385a7465876eb3: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/wals/d55da66aad2140d788704b056e476e99
I20240502 01:54:37.482745 6646 catalog_manager.cc:4830] TS 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083): tablet d55da66aad2140d788704b056e476e99 (table rebalance_test_table_0 [id=145808b6c1a445a19f5a84827cd55658]) successfully deleted
I20240502 01:54:37.483879 11627 raft_consensus.cc:2932] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.485864 8822 raft_consensus.cc:2932] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.487591 6936 raft_consensus.cc:2932] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.488359 12020 raft_consensus.cc:2932] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:37.492749 6653 catalog_manager.cc:5484] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321 reported cstate change: config changed from index 7 to 8, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.495584 6657 catalog_manager.cc:5484] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index 8 to 9, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:37.498638 8956 raft_consensus.cc:2932] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.498665 6801 raft_consensus.cc:2932] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:37.505340 12043 raft_consensus.cc:1050] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c: attempting to promote NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 to VOTER
I20240502 01:54:37.506204 12043 consensus_queue.cc:237] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:37.508256 8691 raft_consensus.cc:2932] T d58f60c0920e4616850cb246acce1fd7 P 3f156b7cfb4f40f481699b257153e5cb [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:37.511925 6801 consensus_queue.cc:237] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:37.513926 8823 raft_consensus.cc:1261] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:37.513830 8824 raft_consensus.cc:2932] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:37.515060 11993 consensus_queue.cc:1035] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.517635 8956 raft_consensus.cc:1261] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:37.518193 11993 consensus_queue.cc:1035] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.522234 11993 raft_consensus.cc:2932] T d1f4c94a105741339ba69e641244ac18 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.526026 8957 raft_consensus.cc:2932] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.530728 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d1f4c94a105741339ba69e641244ac18 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:37.533771 6659 catalog_manager.cc:5484] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc reported cstate change: config changed from index 8 to 9, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.534487 8824 raft_consensus.cc:2932] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.537977 6915 tablet_service.cc:1514] Processing DeleteTablet for tablet d1f4c94a105741339ba69e641244ac18 with delete_type TABLET_DATA_TOMBSTONED (TS 137e2183ff57411791e9af58dfab0a3c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:58498
I20240502 01:54:37.538372 12013 tablet_replica.cc:330] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c: stopping tablet replica
I20240502 01:54:37.538594 12013 raft_consensus.cc:2229] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:37.538930 12013 raft_consensus.cc:2258] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:37.539897 12013 ts_tablet_manager.cc:1875] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:37.545044 12013 ts_tablet_manager.cc:1888] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:37.545742 12013 log.cc:1196] T d1f4c94a105741339ba69e641244ac18 P 137e2183ff57411791e9af58dfab0a3c: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/wals/d1f4c94a105741339ba69e641244ac18
I20240502 01:54:37.546388 6646 catalog_manager.cc:4830] TS 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315): tablet d1f4c94a105741339ba69e641244ac18 (table rebalance_test_table_5 [id=abae7d7cdd254582beebaba1abe05ac1]) successfully deleted
I20240502 01:54:37.547355 10008 raft_consensus.cc:1261] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.547405 9085 raft_consensus.cc:1261] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.548008 12049 consensus_queue.cc:1035] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.548372 12051 consensus_queue.cc:1035] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.549062 7197 raft_consensus.cc:1261] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:37.549669 12051 consensus_queue.cc:1035] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:37.562028 12051 raft_consensus.cc:2932] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.563246 9085 raft_consensus.cc:2932] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.566813 7198 raft_consensus.cc:2932] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.567915 10008 raft_consensus.cc:2932] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.569566 6653 catalog_manager.cc:5484] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c reported cstate change: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:37.575767 6936 consensus_queue.cc:237] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:37.578680 10008 raft_consensus.cc:1261] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:37.578775 9085 raft_consensus.cc:1261] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 137e2183ff57411791e9af58dfab0a3c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:37.579185 12049 consensus_queue.cc:1035] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.579479 12051 consensus_queue.cc:1035] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.582499 12043 raft_consensus.cc:2932] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.585522 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 76fb5e1f69284140a6c6da264a124b3e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:37.586350 9085 raft_consensus.cc:2932] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.587230 6659 catalog_manager.cc:5484] T 76fb5e1f69284140a6c6da264a124b3e P 137e2183ff57411791e9af58dfab0a3c reported cstate change: config changed from index 8 to 9, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "137e2183ff57411791e9af58dfab0a3c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:37.587205 10008 raft_consensus.cc:2932] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:37.591324 7179 tablet_service.cc:1514] Processing DeleteTablet for tablet 76fb5e1f69284140a6c6da264a124b3e with delete_type TABLET_DATA_TOMBSTONED (TS fc21155d36d5416fa3650f0268a20b17 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:40840
I20240502 01:54:37.592451 12066 tablet_replica.cc:330] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17: stopping tablet replica
I20240502 01:54:37.592657 12066 raft_consensus.cc:2229] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:37.593029 12066 raft_consensus.cc:2258] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:37.593768 6801 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4c1d0974a2fd4a548806b6b6560a89f8"
dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48780
I20240502 01:54:37.594043 6801 raft_consensus.cc:595] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:37.594549 12066 ts_tablet_manager.cc:1875] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:37.600700 12066 ts_tablet_manager.cc:1888] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:37.600813 12066 log.cc:1196] T 76fb5e1f69284140a6c6da264a124b3e P fc21155d36d5416fa3650f0268a20b17: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/76fb5e1f69284140a6c6da264a124b3e
I20240502 01:54:37.601553 6645 catalog_manager.cc:4830] TS fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873): tablet 76fb5e1f69284140a6c6da264a124b3e (table rebalance_test_table_5 [id=abae7d7cdd254582beebaba1abe05ac1]) successfully deleted
I20240502 01:54:37.602452 6801 consensus_queue.cc:237] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:37.604431 8822 raft_consensus.cc:1261] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:37.604518 8956 raft_consensus.cc:1261] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:37.605190 11627 consensus_queue.cc:1035] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.605545 11993 consensus_queue.cc:1035] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.608126 11992 raft_consensus.cc:2932] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.611141 8822 raft_consensus.cc:2932] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.610414 8956 raft_consensus.cc:2932] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } }
I20240502 01:54:37.612294 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f03b37fd2f844f61a39daadc698c9b29 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:37.615093 6659 catalog_manager.cc:5484] T f03b37fd2f844f61a39daadc698c9b29 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index 8 to 9, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:37.620491 6915 tablet_service.cc:1514] Processing DeleteTablet for tablet f03b37fd2f844f61a39daadc698c9b29 with delete_type TABLET_DATA_TOMBSTONED (TS 137e2183ff57411791e9af58dfab0a3c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:58498
I20240502 01:54:37.620930 12013 tablet_replica.cc:330] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c: stopping tablet replica
I20240502 01:54:37.621158 12013 raft_consensus.cc:2229] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:37.621481 12013 raft_consensus.cc:2258] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:37.622604 12013 ts_tablet_manager.cc:1875] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:37.625002 11992 raft_consensus.cc:979] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd: : Instructing follower c39b0639d5fb4d9da09bf1d67d617fcc to start an election
I20240502 01:54:37.625174 11999 raft_consensus.cc:1067] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Signalling peer c39b0639d5fb4d9da09bf1d67d617fcc to start an election
I20240502 01:54:37.627048 7197 raft_consensus.cc:1261] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 2 FOLLOWER]: Refusing update from remote peer 3f156b7cfb4f40f481699b257153e5cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:37.627794 11815 consensus_queue.cc:1035] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.628427 8956 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4c1d0974a2fd4a548806b6b6560a89f8"
dest_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc"
from {username='slave'} at 127.31.161.193:60805
I20240502 01:54:37.628705 8956 raft_consensus.cc:482] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:37.628868 12013 ts_tablet_manager.cc:1888] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:37.628918 8956 raft_consensus.cc:3037] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:37.628990 12013 log.cc:1196] T f03b37fd2f844f61a39daadc698c9b29 P 137e2183ff57411791e9af58dfab0a3c: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/wals/f03b37fd2f844f61a39daadc698c9b29
I20240502 01:54:37.631006 6646 catalog_manager.cc:4830] TS 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315): tablet f03b37fd2f844f61a39daadc698c9b29 (table rebalance_test_table_2 [id=3316a56a8d6043c6b05160d6e55a33e0]) successfully deleted
I20240502 01:54:37.635298 8956 raft_consensus.cc:504] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:37.638203 8956 leader_election.cc:288] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc [CANDIDATE]: Term 2 election: Requested vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315), 48866680637c40368e45ba844efb9298 (127.31.161.200:34863)
I20240502 01:54:37.638437 6801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c1d0974a2fd4a548806b6b6560a89f8" candidate_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
I20240502 01:54:37.638698 6801 raft_consensus.cc:3032] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:37.638723 6936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c1d0974a2fd4a548806b6b6560a89f8" candidate_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "137e2183ff57411791e9af58dfab0a3c"
I20240502 01:54:37.638795 6801 raft_consensus.cc:729] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Running, Role: LEADER
I20240502 01:54:37.638921 6936 raft_consensus.cc:3037] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:37.639120 6801 consensus_queue.cc:260] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:37.639618 6801 raft_consensus.cc:3037] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:37.641623 6936 raft_consensus.cc:2454] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39b0639d5fb4d9da09bf1d67d617fcc in term 2.
I20240502 01:54:37.642752 6801 raft_consensus.cc:2454] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39b0639d5fb4d9da09bf1d67d617fcc in term 2.
I20240502 01:54:37.643491 8891 leader_election.cc:302] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 137e2183ff57411791e9af58dfab0a3c, 8214b6c4de6d4aa58a22af551eb8dedd, c39b0639d5fb4d9da09bf1d67d617fcc; no voters:
I20240502 01:54:37.643968 11818 raft_consensus.cc:2781] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:37.644603 11818 raft_consensus.cc:686] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc [term 2 LEADER]: Becoming Leader. State: Replica: c39b0639d5fb4d9da09bf1d67d617fcc, State: Running, Role: LEADER
I20240502 01:54:37.645246 11818 consensus_queue.cc:237] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:37.649214 6654 catalog_manager.cc:5484] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc reported cstate change: term changed from 1 to 2, leader changed from 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) to c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199). New cstate: current_term: 2 leader_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:37.664070 9085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c1d0974a2fd4a548806b6b6560a89f8" candidate_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "48866680637c40368e45ba844efb9298"
I20240502 01:54:37.664312 9085 raft_consensus.cc:3037] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:37.667603 9085 raft_consensus.cc:2454] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39b0639d5fb4d9da09bf1d67d617fcc in term 2.
I20240502 01:54:37.683198 7066 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d58f60c0920e4616850cb246acce1fd7"
dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55482
I20240502 01:54:37.683429 7066 raft_consensus.cc:595] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 2 LEADER]: Received request to transfer leadership
I20240502 01:54:37.707908 6936 raft_consensus.cc:1261] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c [term 2 FOLLOWER]: Refusing update from remote peer 3f156b7cfb4f40f481699b257153e5cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:37.708575 11815 consensus_queue.cc:1035] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.721482 9085 raft_consensus.cc:1261] T 70ce4bbee414402aa97c9f3914b9d0c6 P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Refusing update from remote peer 3f156b7cfb4f40f481699b257153e5cb: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:37.727242 6935 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8960ca8e5d26429589abc4a154311983"
dest_uuid: "137e2183ff57411791e9af58dfab0a3c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48848
I20240502 01:54:37.727542 6935 raft_consensus.cc:595] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:37.731118 12024 consensus_queue.cc:1035] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:37.750648 7066 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "640b6474d1834321881094ed0b11b857"
dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55488
I20240502 01:54:37.750861 7066 raft_consensus.cc:595] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:37.777925 7066 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f6a165cbc26f4defb70866779f83bfdc"
dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55506
I20240502 01:54:37.778128 7066 raft_consensus.cc:595] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:54:37.782562 12020 raft_consensus.cc:979] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3: : Instructing follower fc21155d36d5416fa3650f0268a20b17 to start an election
I20240502 01:54:37.782761 12021 raft_consensus.cc:1067] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Signalling peer fc21155d36d5416fa3650f0268a20b17 to start an election
I20240502 01:54:37.783550 7197 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "640b6474d1834321881094ed0b11b857"
dest_uuid: "fc21155d36d5416fa3650f0268a20b17"
from {username='slave'} at 127.31.161.195:47719
I20240502 01:54:37.783820 7197 raft_consensus.cc:482] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:37.783962 7197 raft_consensus.cc:3037] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:37.786605 7197 raft_consensus.cc:504] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:37.787907 7066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640b6474d1834321881094ed0b11b857" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
I20240502 01:54:37.787997 8691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640b6474d1834321881094ed0b11b857" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "3f156b7cfb4f40f481699b257153e5cb"
I20240502 01:54:37.788125 7066 raft_consensus.cc:3032] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:37.788213 7066 raft_consensus.cc:729] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Running, Role: LEADER
I20240502 01:54:37.788327 8691 raft_consensus.cc:3037] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:37.788558 7066 consensus_queue.cc:260] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:37.788983 7066 raft_consensus.cc:3037] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:37.791106 8691 raft_consensus.cc:2454] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 2.
I20240502 01:54:37.791553 7066 raft_consensus.cc:2454] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 2.
I20240502 01:54:37.791889 9085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640b6474d1834321881094ed0b11b857" candidate_uuid: "fc21155d36d5416fa3650f0268a20b17" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "48866680637c40368e45ba844efb9298"
I20240502 01:54:37.792071 9085 raft_consensus.cc:3037] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:37.793426 7134 leader_election.cc:302] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 3f156b7cfb4f40f481699b257153e5cb, fc21155d36d5416fa3650f0268a20b17; no voters:
I20240502 01:54:37.793747 11280 raft_consensus.cc:2781] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:37.793890 11280 raft_consensus.cc:686] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [term 2 LEADER]: Becoming Leader. State: Replica: fc21155d36d5416fa3650f0268a20b17, State: Running, Role: LEADER
I20240502 01:54:37.794142 9085 raft_consensus.cc:2454] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc21155d36d5416fa3650f0268a20b17 in term 2.
I20240502 01:54:37.794198 11280 consensus_queue.cc:237] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:37.794508 7197 leader_election.cc:288] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [CANDIDATE]: Term 2 election: Requested vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 48866680637c40368e45ba844efb9298 (127.31.161.200:34863), 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197:44451)
I20240502 01:54:37.797390 6657 catalog_manager.cc:5484] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: term changed from 1 to 2, leader changed from 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) to fc21155d36d5416fa3650f0268a20b17 (127.31.161.196). New cstate: current_term: 2 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:37.818280 12028 raft_consensus.cc:979] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3: : Instructing follower 48866680637c40368e45ba844efb9298 to start an election
I20240502 01:54:37.818454 12028 raft_consensus.cc:1067] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Signalling peer 48866680637c40368e45ba844efb9298 to start an election
I20240502 01:54:37.819190 9085 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f6a165cbc26f4defb70866779f83bfdc"
dest_uuid: "48866680637c40368e45ba844efb9298"
from {username='slave'} at 127.31.161.195:43227
I20240502 01:54:37.819500 9085 raft_consensus.cc:482] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:37.819638 9085 raft_consensus.cc:3037] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:37.820837 12043 raft_consensus.cc:979] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c: : Instructing follower 8214b6c4de6d4aa58a22af551eb8dedd to start an election
I20240502 01:54:37.820991 12043 raft_consensus.cc:1067] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Signalling peer 8214b6c4de6d4aa58a22af551eb8dedd to start an election
I20240502 01:54:37.821823 9085 raft_consensus.cc:504] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:37.821943 6801 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8960ca8e5d26429589abc4a154311983"
dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
from {username='slave'} at 127.31.161.194:38239
I20240502 01:54:37.822216 6801 raft_consensus.cc:482] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:37.822338 6801 raft_consensus.cc:3037] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:37.823474 9085 leader_election.cc:288] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [CANDIDATE]: Term 2 election: Requested vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199:35245)
I20240502 01:54:37.824290 7066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a165cbc26f4defb70866779f83bfdc" candidate_uuid: "48866680637c40368e45ba844efb9298" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
I20240502 01:54:37.824515 7066 raft_consensus.cc:3032] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:37.824599 7066 raft_consensus.cc:729] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Running, Role: LEADER
I20240502 01:54:37.824905 7066 consensus_queue.cc:260] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:37.825337 7066 raft_consensus.cc:3037] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:37.826601 6803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a165cbc26f4defb70866779f83bfdc" candidate_uuid: "48866680637c40368e45ba844efb9298" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
I20240502 01:54:37.826803 6803 raft_consensus.cc:3037] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:37.827723 6801 raft_consensus.cc:504] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:37.827872 7066 raft_consensus.cc:2454] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48866680637c40368e45ba844efb9298 in term 2.
I20240502 01:54:37.831281 6936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8960ca8e5d26429589abc4a154311983" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "137e2183ff57411791e9af58dfab0a3c"
I20240502 01:54:37.831537 6936 raft_consensus.cc:3032] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:54:37.831646 6936 raft_consensus.cc:729] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 137e2183ff57411791e9af58dfab0a3c, State: Running, Role: LEADER
I20240502 01:54:37.831975 6936 consensus_queue.cc:260] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:37.832490 6936 raft_consensus.cc:3037] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:37.830083 6803 raft_consensus.cc:2454] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48866680637c40368e45ba844efb9298 in term 2.
I20240502 01:54:37.833825 8956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6a165cbc26f4defb70866779f83bfdc" candidate_uuid: "48866680637c40368e45ba844efb9298" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc"
I20240502 01:54:37.834041 9021 leader_election.cc:302] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 48866680637c40368e45ba844efb9298, 8214b6c4de6d4aa58a22af551eb8dedd; no voters:
I20240502 01:54:37.834136 8956 raft_consensus.cc:3037] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:37.834355 6801 leader_election.cc:288] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 2 election: Requested vote from peers 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315), 48866680637c40368e45ba844efb9298 (127.31.161.200:34863), fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201:43211)
I20240502 01:54:37.834825 11746 raft_consensus.cc:2781] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:37.835155 6936 raft_consensus.cc:2454] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 2.
I20240502 01:54:37.835815 9086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8960ca8e5d26429589abc4a154311983" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "48866680637c40368e45ba844efb9298"
I20240502 01:54:37.836027 9086 raft_consensus.cc:3037] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:37.836081 11746 raft_consensus.cc:686] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [term 2 LEADER]: Becoming Leader. State: Replica: 48866680637c40368e45ba844efb9298, State: Running, Role: LEADER
I20240502 01:54:37.836400 8956 raft_consensus.cc:2454] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48866680637c40368e45ba844efb9298 in term 2.
I20240502 01:54:37.836532 11746 consensus_queue.cc:237] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:37.839625 9086 raft_consensus.cc:2454] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 2.
I20240502 01:54:37.843636 6657 catalog_manager.cc:5484] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 reported cstate change: term changed from 1 to 2, leader changed from 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) to 48866680637c40368e45ba844efb9298 (127.31.161.200). New cstate: current_term: 2 leader_uuid: "48866680637c40368e45ba844efb9298" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:37.846112 6738 leader_election.cc:302] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 137e2183ff57411791e9af58dfab0a3c, 48866680637c40368e45ba844efb9298, 8214b6c4de6d4aa58a22af551eb8dedd; no voters:
I20240502 01:54:37.847251 11992 raft_consensus.cc:2781] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:37.847440 11992 raft_consensus.cc:686] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [term 2 LEADER]: Becoming Leader. State: Replica: 8214b6c4de6d4aa58a22af551eb8dedd, State: Running, Role: LEADER
I20240502 01:54:37.847733 11992 consensus_queue.cc:237] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:37.850791 6657 catalog_manager.cc:5484] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: term changed from 1 to 2, leader changed from 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) to 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193). New cstate: current_term: 2 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:37.855928 6801 consensus_queue.cc:237] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:37.858354 8691 raft_consensus.cc:1261] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:37.858278 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8960ca8e5d26429589abc4a154311983" candidate_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "fdfc60a29ea9465c88ee45c3c332caa1"
I20240502 01:54:37.858657 7197 raft_consensus.cc:1261] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:37.858640 10008 raft_consensus.cc:3037] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:37.859177 11999 consensus_queue.cc:1035] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:37.859637 11999 consensus_queue.cc:1035] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:37.861027 10008 raft_consensus.cc:2454] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8214b6c4de6d4aa58a22af551eb8dedd in term 2.
W20240502 01:54:37.862066 6738 consensus_peers.cc:485] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd -> Peer fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201:43211): Couldn't send request to peer fdfc60a29ea9465c88ee45c3c332caa1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 945224f5cff74edb9447bebc743e5086. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:37.862186 11992 raft_consensus.cc:2932] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:37.863155 7197 raft_consensus.cc:2932] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:37.865535 8691 raft_consensus.cc:2932] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:37.868659 6654 catalog_manager.cc:5484] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:37.885797 7066 consensus_queue.cc:237] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:37.888202 8956 raft_consensus.cc:1261] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:37.888191 6801 raft_consensus.cc:1261] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:37.888736 12028 consensus_queue.cc:1035] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:37.889035 12020 consensus_queue.cc:1035] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:37.891732 12028 raft_consensus.cc:2932] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:37.892432 6801 raft_consensus.cc:2932] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:37.898046 6659 catalog_manager.cc:5484] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 01:54:37.903558 7003 consensus_peers.cc:485] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 -> Peer fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201:43211): Couldn't send request to peer fdfc60a29ea9465c88ee45c3c332caa1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43e6e5b3435a446499106efe7baf92aa. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:37.905107 8956 raft_consensus.cc:2932] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:37.928628 6801 consensus_queue.cc:237] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:37.929194 12020 raft_consensus.cc:979] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3: : Instructing follower 9e47e5c098b848c9abe1559cbcbb6321 to start an election
I20240502 01:54:37.929355 12020 raft_consensus.cc:1067] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 2 LEADER]: Signalling peer 9e47e5c098b848c9abe1559cbcbb6321 to start an election
I20240502 01:54:37.930087 8822 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d58f60c0920e4616850cb246acce1fd7"
dest_uuid: "9e47e5c098b848c9abe1559cbcbb6321"
from {username='slave'} at 127.31.161.195:52447
I20240502 01:54:37.930383 8822 raft_consensus.cc:482] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:54:37.930541 8822 raft_consensus.cc:3037] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:37.931761 8823 raft_consensus.cc:1261] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 01:54:37.932299 6738 consensus_peers.cc:485] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd -> Peer fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201:43211): Couldn't send request to peer fdfc60a29ea9465c88ee45c3c332caa1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a14c3c945b724df5ac3f917cef3186e0. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:37.932657 11999 consensus_queue.cc:1035] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:37.933228 8822 raft_consensus.cc:504] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:37.935117 8822 leader_election.cc:288] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [CANDIDATE]: Term 3 election: Requested vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197:44451)
I20240502 01:54:37.935478 6801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58f60c0920e4616850cb246acce1fd7" candidate_uuid: "9e47e5c098b848c9abe1559cbcbb6321" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "8214b6c4de6d4aa58a22af551eb8dedd"
I20240502 01:54:37.935681 6801 raft_consensus.cc:3037] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:37.935853 7066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58f60c0920e4616850cb246acce1fd7" candidate_uuid: "9e47e5c098b848c9abe1559cbcbb6321" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "11c097d7d32a4c22a6385a7465876eb3"
I20240502 01:54:37.936028 6936 raft_consensus.cc:1261] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:37.936064 7066 raft_consensus.cc:3032] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 01:54:37.936152 7066 raft_consensus.cc:729] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 11c097d7d32a4c22a6385a7465876eb3, State: Running, Role: LEADER
I20240502 01:54:37.936100 11999 raft_consensus.cc:2932] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:37.936411 7066 consensus_queue.cc:260] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:37.936851 7066 raft_consensus.cc:3037] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:37.938679 6801 raft_consensus.cc:2454] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e47e5c098b848c9abe1559cbcbb6321 in term 3.
I20240502 01:54:37.939472 8822 raft_consensus.cc:2932] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:37.940074 11993 consensus_queue.cc:1035] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:37.940673 7066 raft_consensus.cc:2454] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e47e5c098b848c9abe1559cbcbb6321 in term 3.
I20240502 01:54:37.942800 6657 catalog_manager.cc:5484] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:37.947577 8759 leader_election.cc:302] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 11c097d7d32a4c22a6385a7465876eb3, 8214b6c4de6d4aa58a22af551eb8dedd, 9e47e5c098b848c9abe1559cbcbb6321; no voters:
I20240502 01:54:37.948033 11768 raft_consensus.cc:2781] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 01:54:37.960856 11768 raft_consensus.cc:686] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [term 3 LEADER]: Becoming Leader. State: Replica: 9e47e5c098b848c9abe1559cbcbb6321, State: Running, Role: LEADER
I20240502 01:54:37.961488 11768 consensus_queue.cc:237] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:37.962931 8691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d58f60c0920e4616850cb246acce1fd7" candidate_uuid: "9e47e5c098b848c9abe1559cbcbb6321" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: true dest_uuid: "3f156b7cfb4f40f481699b257153e5cb"
I20240502 01:54:37.963140 8691 raft_consensus.cc:3037] T d58f60c0920e4616850cb246acce1fd7 P 3f156b7cfb4f40f481699b257153e5cb [term 2 FOLLOWER]: Advancing to term 3
I20240502 01:54:37.965446 8691 raft_consensus.cc:2454] T d58f60c0920e4616850cb246acce1fd7 P 3f156b7cfb4f40f481699b257153e5cb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e47e5c098b848c9abe1559cbcbb6321 in term 3.
I20240502 01:54:37.966084 6936 raft_consensus.cc:2932] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:37.968497 6659 catalog_manager.cc:5484] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 reported cstate change: term changed from 2 to 3, leader changed from 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) to 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198). New cstate: current_term: 3 leader_uuid: "9e47e5c098b848c9abe1559cbcbb6321" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:37.974941 7066 consensus_queue.cc:237] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:37.977615 7197 raft_consensus.cc:1261] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:37.978087 8822 raft_consensus.cc:1261] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:37.978118 12020 consensus_queue.cc:1035] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 01:54:37.978757 7003 consensus_peers.cc:485] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 -> Peer fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201:43211): Couldn't send request to peer fdfc60a29ea9465c88ee45c3c332caa1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b10e180c11c4cd4a308fc7d311840c2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:37.979044 12020 consensus_queue.cc:1035] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:37.981846 12028 raft_consensus.cc:2932] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:37.985693 7197 raft_consensus.cc:2932] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:37.987085 6659 catalog_manager.cc:5484] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:37.987813 8822 raft_consensus.cc:2932] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:38.219174 8956 raft_consensus.cc:1261] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc [term 2 FOLLOWER]: Refusing update from remote peer 48866680637c40368e45ba844efb9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:38.220623 11746 consensus_queue.cc:1035] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:38.226591 8691 consensus_queue.cc:237] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } }
I20240502 01:54:38.228345 9086 raft_consensus.cc:1261] T 70ce4bbee414402aa97c9f3914b9d0c6 P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Refusing update from remote peer 3f156b7cfb4f40f481699b257153e5cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:54:38.228610 6936 raft_consensus.cc:1261] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c [term 2 FOLLOWER]: Refusing update from remote peer 3f156b7cfb4f40f481699b257153e5cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:54:38.228880 6935 raft_consensus.cc:1261] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c [term 2 FOLLOWER]: Refusing update from remote peer c39b0639d5fb4d9da09bf1d67d617fcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:38.229204 12024 consensus_queue.cc:1035] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:38.229518 11815 consensus_queue.cc:1035] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:38.229849 11818 consensus_queue.cc:1035] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:38.233954 11815 raft_consensus.cc:2932] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:38.237711 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 70ce4bbee414402aa97c9f3914b9d0c6 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:38.237746 9085 raft_consensus.cc:1261] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:38.238643 11903 consensus_queue.cc:1035] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:38.238598 6935 raft_consensus.cc:2932] T 70ce4bbee414402aa97c9f3914b9d0c6 P 137e2183ff57411791e9af58dfab0a3c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:38.239274 9086 raft_consensus.cc:2932] T 70ce4bbee414402aa97c9f3914b9d0c6 P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } }
I20240502 01:54:38.243984 6659 catalog_manager.cc:5484] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb reported cstate change: config changed from index 8 to 10, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New cstate: current_term: 2 leader_uuid: "3f156b7cfb4f40f481699b257153e5cb" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:38.246510 9085 raft_consensus.cc:1261] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Refusing update from remote peer c39b0639d5fb4d9da09bf1d67d617fcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:38.247042 11818 consensus_queue.cc:1035] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:38.249356 7179 tablet_service.cc:1514] Processing DeleteTablet for tablet 70ce4bbee414402aa97c9f3914b9d0c6 with delete_type TABLET_DATA_TOMBSTONED (TS fc21155d36d5416fa3650f0268a20b17 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:40840
I20240502 01:54:38.256319 12339 tablet_replica.cc:330] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17: stopping tablet replica
I20240502 01:54:38.256583 12339 raft_consensus.cc:2229] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:38.256860 12339 raft_consensus.cc:2258] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:38.258002 12339 ts_tablet_manager.cc:1875] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:38.263659 6801 raft_consensus.cc:1261] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 2 FOLLOWER]: Refusing update from remote peer c39b0639d5fb4d9da09bf1d67d617fcc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:38.264711 11818 consensus_queue.cc:1035] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:38.265367 12339 ts_tablet_manager.cc:1888] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 01:54:38.265545 12339 log.cc:1196] T 70ce4bbee414402aa97c9f3914b9d0c6 P fc21155d36d5416fa3650f0268a20b17: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/70ce4bbee414402aa97c9f3914b9d0c6
I20240502 01:54:38.266439 6645 catalog_manager.cc:4830] TS fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873): tablet 70ce4bbee414402aa97c9f3914b9d0c6 (table rebalance_test_table_1 [id=9a08c31578ae4224915289c7c289ee7a]) successfully deleted
I20240502 01:54:38.280949 9085 raft_consensus.cc:1261] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:38.281538 11999 consensus_queue.cc:1035] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:38.284842 12347 ts_tablet_manager.cc:902] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1: Initiating tablet copy from peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:38.286171 12347 tablet_copy_client.cc:308] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.193:45489
I20240502 01:54:38.292419 6825 tablet_copy_service.cc:140] P 8214b6c4de6d4aa58a22af551eb8dedd: Received BeginTabletCopySession request for tablet 945224f5cff74edb9447bebc743e5086 from peer fdfc60a29ea9465c88ee45c3c332caa1 ({username='slave'} at 127.31.161.201:56759)
I20240502 01:54:38.292570 6825 tablet_copy_service.cc:161] P 8214b6c4de6d4aa58a22af551eb8dedd: Beginning new tablet copy session on tablet 945224f5cff74edb9447bebc743e5086 from peer fdfc60a29ea9465c88ee45c3c332caa1 at {username='slave'} at 127.31.161.201:56759: session id = fdfc60a29ea9465c88ee45c3c332caa1-945224f5cff74edb9447bebc743e5086
I20240502 01:54:38.294467 6825 tablet_copy_source_session.cc:205] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:38.295681 12347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 945224f5cff74edb9447bebc743e5086. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.300136 12347 tablet_copy_client.cc:783] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:38.300354 12347 tablet_copy_client.cc:647] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:38.301715 12351 tablet_copy_client.cc:1002] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:38.302724 8956 consensus_queue.cc:237] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:38.304653 9085 raft_consensus.cc:1261] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Refusing update from remote peer c39b0639d5fb4d9da09bf1d67d617fcc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:54:38.305181 6935 raft_consensus.cc:1261] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c [term 2 FOLLOWER]: Refusing update from remote peer c39b0639d5fb4d9da09bf1d67d617fcc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:54:38.305509 12343 consensus_queue.cc:1035] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:38.305779 11818 consensus_queue.cc:1035] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:38.308250 12340 raft_consensus.cc:2932] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:38.309177 6935 raft_consensus.cc:2932] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:38.309216 9085 raft_consensus.cc:2932] T 4c1d0974a2fd4a548806b6b6560a89f8 P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:38.312431 10009 raft_consensus.cc:1261] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [term 2 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:38.314195 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4c1d0974a2fd4a548806b6b6560a89f8 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:38.314163 11999 consensus_queue.cc:1035] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:38.314864 6659 catalog_manager.cc:5484] T 4c1d0974a2fd4a548806b6b6560a89f8 P 137e2183ff57411791e9af58dfab0a3c reported cstate change: config changed from index 8 to 10, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New cstate: current_term: 2 leader_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:38.320276 6782 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c1d0974a2fd4a548806b6b6560a89f8 with delete_type TABLET_DATA_TOMBSTONED (TS 8214b6c4de6d4aa58a22af551eb8dedd not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:35388
I20240502 01:54:38.324779 6935 raft_consensus.cc:1261] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 2 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:38.326239 11999 consensus_queue.cc:1035] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:38.328292 6801 consensus_queue.cc:237] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:38.330169 9086 raft_consensus.cc:1261] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:54:38.330945 11992 consensus_queue.cc:1035] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:38.331753 12355 tablet_replica.cc:330] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd: stopping tablet replica
I20240502 01:54:38.331991 12355 raft_consensus.cc:2229] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:38.332245 12355 raft_consensus.cc:2258] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:38.333140 12355 ts_tablet_manager.cc:1875] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:38.336611 11999 raft_consensus.cc:2932] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:38.337422 10008 raft_consensus.cc:1261] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [term 2 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:54:38.337301 9086 raft_consensus.cc:2932] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:38.338625 12355 ts_tablet_manager.cc:1888] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 01:54:38.338752 12355 log.cc:1196] T 4c1d0974a2fd4a548806b6b6560a89f8 P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/4c1d0974a2fd4a548806b6b6560a89f8
I20240502 01:54:38.339515 6645 catalog_manager.cc:4830] TS 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): tablet 4c1d0974a2fd4a548806b6b6560a89f8 (table rebalance_test_table_9 [id=5b2a4358eba544959466c46759744194]) successfully deleted
I20240502 01:54:38.340567 11627 consensus_queue.cc:1035] T 8960ca8e5d26429589abc4a154311983 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:38.341329 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8960ca8e5d26429589abc4a154311983 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:38.342010 6653 catalog_manager.cc:5484] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 reported cstate change: config changed from index 8 to 10, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New cstate: current_term: 2 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:38.341977 10009 raft_consensus.cc:2932] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:38.346896 6915 tablet_service.cc:1514] Processing DeleteTablet for tablet 8960ca8e5d26429589abc4a154311983 with delete_type TABLET_DATA_TOMBSTONED (TS 137e2183ff57411791e9af58dfab0a3c not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:58498
I20240502 01:54:38.349650 12358 tablet_replica.cc:330] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c: stopping tablet replica
I20240502 01:54:38.350356 12358 raft_consensus.cc:2229] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:38.350638 12358 raft_consensus.cc:2258] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:38.351737 12358 ts_tablet_manager.cc:1875] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:38.357204 8691 raft_consensus.cc:1261] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb [term 2 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:38.357833 11903 consensus_queue.cc:1035] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:38.361402 12358 ts_tablet_manager.cc:1888] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 01:54:38.361524 12358 log.cc:1196] T 8960ca8e5d26429589abc4a154311983 P 137e2183ff57411791e9af58dfab0a3c: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/wals/8960ca8e5d26429589abc4a154311983
I20240502 01:54:38.362301 6646 catalog_manager.cc:4830] TS 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315): tablet 8960ca8e5d26429589abc4a154311983 (table rebalance_test_table_6 [id=981b993f808c4320845154760fb7959f]) successfully deleted
I20240502 01:54:38.364606 7066 raft_consensus.cc:1261] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 2 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:38.366940 12360 consensus_queue.cc:1035] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:38.369452 7197 consensus_queue.cc:237] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:38.371601 9086 raft_consensus.cc:1261] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:54:38.371650 8692 raft_consensus.cc:1261] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb [term 2 FOLLOWER]: Refusing update from remote peer fc21155d36d5416fa3650f0268a20b17: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:54:38.372438 12359 consensus_queue.cc:1035] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:38.372735 11903 consensus_queue.cc:1035] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:38.375072 7067 raft_consensus.cc:1261] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 2 FOLLOWER]: Refusing update from remote peer 48866680637c40368e45ba844efb9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:38.375612 11746 consensus_queue.cc:1035] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:38.375751 11903 raft_consensus.cc:2932] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:38.376039 9086 raft_consensus.cc:2932] T 640b6474d1834321881094ed0b11b857 P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:38.376571 8692 raft_consensus.cc:2932] T 640b6474d1834321881094ed0b11b857 P 3f156b7cfb4f40f481699b257153e5cb [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:38.379606 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 640b6474d1834321881094ed0b11b857 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:38.381852 6659 catalog_manager.cc:5484] T 640b6474d1834321881094ed0b11b857 P fc21155d36d5416fa3650f0268a20b17 reported cstate change: config changed from index 8 to 10, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New cstate: current_term: 2 leader_uuid: "fc21155d36d5416fa3650f0268a20b17" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:38.385716 7047 tablet_service.cc:1514] Processing DeleteTablet for tablet 640b6474d1834321881094ed0b11b857 with delete_type TABLET_DATA_TOMBSTONED (TS 11c097d7d32a4c22a6385a7465876eb3 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:59762
I20240502 01:54:38.386329 6801 raft_consensus.cc:1261] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd [term 2 FOLLOWER]: Refusing update from remote peer 48866680637c40368e45ba844efb9298: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 01:54:38.387013 11746 consensus_queue.cc:1035] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:38.402944 12368 tablet_replica.cc:330] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3: stopping tablet replica
I20240502 01:54:38.403232 12368 raft_consensus.cc:2229] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:38.403513 12368 raft_consensus.cc:2258] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:38.404531 12368 ts_tablet_manager.cc:1875] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:38.406756 12370 ts_tablet_manager.cc:902] T 43e6e5b3435a446499106efe7baf92aa P fdfc60a29ea9465c88ee45c3c332caa1: Initiating tablet copy from peer 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:38.410778 12368 ts_tablet_manager.cc:1888] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 01:54:38.410884 12368 log.cc:1196] T 640b6474d1834321881094ed0b11b857 P 11c097d7d32a4c22a6385a7465876eb3: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/wals/640b6474d1834321881094ed0b11b857
I20240502 01:54:38.411518 6646 catalog_manager.cc:4830] TS 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083): tablet 640b6474d1834321881094ed0b11b857 (table rebalance_test_table_4 [id=9231a7576c804559bc4a4c54496d5223]) successfully deleted
I20240502 01:54:38.412405 12370 tablet_copy_client.cc:308] T 43e6e5b3435a446499106efe7baf92aa P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:46083
I20240502 01:54:38.418128 7089 tablet_copy_service.cc:140] P 11c097d7d32a4c22a6385a7465876eb3: Received BeginTabletCopySession request for tablet 43e6e5b3435a446499106efe7baf92aa from peer fdfc60a29ea9465c88ee45c3c332caa1 ({username='slave'} at 127.31.161.201:57973)
I20240502 01:54:38.418282 7089 tablet_copy_service.cc:161] P 11c097d7d32a4c22a6385a7465876eb3: Beginning new tablet copy session on tablet 43e6e5b3435a446499106efe7baf92aa from peer fdfc60a29ea9465c88ee45c3c332caa1 at {username='slave'} at 127.31.161.201:57973: session id = fdfc60a29ea9465c88ee45c3c332caa1-43e6e5b3435a446499106efe7baf92aa
I20240502 01:54:38.420493 7089 tablet_copy_source_session.cc:205] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:38.421716 12370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e6e5b3435a446499106efe7baf92aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.425859 12370 tablet_copy_client.cc:783] T 43e6e5b3435a446499106efe7baf92aa P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:38.426093 12370 tablet_copy_client.cc:647] T 43e6e5b3435a446499106efe7baf92aa P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:38.427508 12375 tablet_copy_client.cc:1002] T 43e6e5b3435a446499106efe7baf92aa P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:38.497785 8692 raft_consensus.cc:1261] T d58f60c0920e4616850cb246acce1fd7 P 3f156b7cfb4f40f481699b257153e5cb [term 3 FOLLOWER]: Refusing update from remote peer 9e47e5c098b848c9abe1559cbcbb6321: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 01:54:38.498669 12411 consensus_queue.cc:1035] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:38.501474 6801 raft_consensus.cc:1261] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 3 FOLLOWER]: Refusing update from remote peer 9e47e5c098b848c9abe1559cbcbb6321: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 01:54:38.501991 12412 consensus_queue.cc:1035] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:38.514339 7067 raft_consensus.cc:1261] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 3 FOLLOWER]: Refusing update from remote peer 9e47e5c098b848c9abe1559cbcbb6321: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 01:54:38.515823 12412 consensus_queue.cc:1035] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:38.543725 8822 consensus_queue.cc:237] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } }
I20240502 01:54:38.545893 8692 raft_consensus.cc:1261] T d58f60c0920e4616850cb246acce1fd7 P 3f156b7cfb4f40f481699b257153e5cb [term 3 FOLLOWER]: Refusing update from remote peer 9e47e5c098b848c9abe1559cbcbb6321: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 01:54:38.546077 6801 raft_consensus.cc:1261] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 3 FOLLOWER]: Refusing update from remote peer 9e47e5c098b848c9abe1559cbcbb6321: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 01:54:38.546545 12419 consensus_queue.cc:1035] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:54:38.546887 12420 consensus_queue.cc:1035] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:54:38.549254 12420 raft_consensus.cc:2932] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:38.549894 6801 raft_consensus.cc:2932] T d58f60c0920e4616850cb246acce1fd7 P 8214b6c4de6d4aa58a22af551eb8dedd [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:38.553753 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d58f60c0920e4616850cb246acce1fd7 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:38.555297 6659 catalog_manager.cc:5484] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 reported cstate change: config changed from index 9 to 11, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New cstate: current_term: 3 leader_uuid: "9e47e5c098b848c9abe1559cbcbb6321" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:38.555239 8692 raft_consensus.cc:2932] T d58f60c0920e4616850cb246acce1fd7 P 3f156b7cfb4f40f481699b257153e5cb [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 9 to 11, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } }
I20240502 01:54:38.559396 7047 tablet_service.cc:1514] Processing DeleteTablet for tablet d58f60c0920e4616850cb246acce1fd7 with delete_type TABLET_DATA_TOMBSTONED (TS 11c097d7d32a4c22a6385a7465876eb3 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:59762
I20240502 01:54:38.559793 12368 tablet_replica.cc:330] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3: stopping tablet replica
I20240502 01:54:38.560011 12368 raft_consensus.cc:2229] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:38.560245 12368 raft_consensus.cc:2258] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:38.565143 12433 ts_tablet_manager.cc:902] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1: Initiating tablet copy from peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:38.565186 12368 ts_tablet_manager.cc:1875] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:38.571005 12368 ts_tablet_manager.cc:1888] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.10
I20240502 01:54:38.571146 12368 log.cc:1196] T d58f60c0920e4616850cb246acce1fd7 P 11c097d7d32a4c22a6385a7465876eb3: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/wals/d58f60c0920e4616850cb246acce1fd7
I20240502 01:54:38.571954 6646 catalog_manager.cc:4830] TS 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083): tablet d58f60c0920e4616850cb246acce1fd7 (table rebalance_test_table_7 [id=4d5b7e86d6b24299a377cfaff5756f47]) successfully deleted
I20240502 01:54:38.579797 12433 tablet_copy_client.cc:308] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.193:45489
I20240502 01:54:38.580686 6825 tablet_copy_service.cc:140] P 8214b6c4de6d4aa58a22af551eb8dedd: Received BeginTabletCopySession request for tablet a14c3c945b724df5ac3f917cef3186e0 from peer fdfc60a29ea9465c88ee45c3c332caa1 ({username='slave'} at 127.31.161.201:56759)
I20240502 01:54:38.580854 6825 tablet_copy_service.cc:161] P 8214b6c4de6d4aa58a22af551eb8dedd: Beginning new tablet copy session on tablet a14c3c945b724df5ac3f917cef3186e0 from peer fdfc60a29ea9465c88ee45c3c332caa1 at {username='slave'} at 127.31.161.201:56759: session id = fdfc60a29ea9465c88ee45c3c332caa1-a14c3c945b724df5ac3f917cef3186e0
I20240502 01:54:38.583667 6825 tablet_copy_source_session.cc:205] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:38.584853 12433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a14c3c945b724df5ac3f917cef3186e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.586720 6801 consensus_queue.cc:237] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:38.589272 9086 raft_consensus.cc:1261] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:38.589407 6935 raft_consensus.cc:1261] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:54:38.589932 11992 consensus_queue.cc:1035] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 01:54:38.590279 6738 consensus_peers.cc:485] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd -> Peer fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201:43211): Couldn't send request to peer fdfc60a29ea9465c88ee45c3c332caa1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39f0a7df4f624eee81c25b41d19d1740. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:38.590415 12433 tablet_copy_client.cc:783] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:38.590569 11992 consensus_queue.cc:1035] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:54:38.590673 12433 tablet_copy_client.cc:647] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:38.593995 11627 raft_consensus.cc:2932] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:38.598721 12437 tablet_copy_client.cc:1002] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:38.598734 6935 raft_consensus.cc:2932] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:38.605190 6653 catalog_manager.cc:5484] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c reported cstate change: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:38.606204 12444 ts_tablet_manager.cc:902] T 4b10e180c11c4cd4a308fc7d311840c2 P fdfc60a29ea9465c88ee45c3c332caa1: Initiating tablet copy from peer 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
I20240502 01:54:38.608863 9086 raft_consensus.cc:2932] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } }
I20240502 01:54:38.618971 12444 tablet_copy_client.cc:308] T 4b10e180c11c4cd4a308fc7d311840c2 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:46083
I20240502 01:54:38.619817 7089 tablet_copy_service.cc:140] P 11c097d7d32a4c22a6385a7465876eb3: Received BeginTabletCopySession request for tablet 4b10e180c11c4cd4a308fc7d311840c2 from peer fdfc60a29ea9465c88ee45c3c332caa1 ({username='slave'} at 127.31.161.201:57973)
I20240502 01:54:38.619967 7089 tablet_copy_service.cc:161] P 11c097d7d32a4c22a6385a7465876eb3: Beginning new tablet copy session on tablet 4b10e180c11c4cd4a308fc7d311840c2 from peer fdfc60a29ea9465c88ee45c3c332caa1 at {username='slave'} at 127.31.161.201:57973: session id = fdfc60a29ea9465c88ee45c3c332caa1-4b10e180c11c4cd4a308fc7d311840c2
I20240502 01:54:38.621925 7089 tablet_copy_source_session.cc:205] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:38.623245 12444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b10e180c11c4cd4a308fc7d311840c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.629086 12444 tablet_copy_client.cc:783] T 4b10e180c11c4cd4a308fc7d311840c2 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:38.629376 12444 tablet_copy_client.cc:647] T 4b10e180c11c4cd4a308fc7d311840c2 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:38.631111 12454 tablet_copy_client.cc:1002] T 4b10e180c11c4cd4a308fc7d311840c2 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:38.965822 12524 ts_tablet_manager.cc:902] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1: Initiating tablet copy from peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
I20240502 01:54:38.969151 12524 tablet_copy_client.cc:308] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.193:45489
I20240502 01:54:38.969853 6825 tablet_copy_service.cc:140] P 8214b6c4de6d4aa58a22af551eb8dedd: Received BeginTabletCopySession request for tablet 39f0a7df4f624eee81c25b41d19d1740 from peer fdfc60a29ea9465c88ee45c3c332caa1 ({username='slave'} at 127.31.161.201:56759)
I20240502 01:54:38.970067 6825 tablet_copy_service.cc:161] P 8214b6c4de6d4aa58a22af551eb8dedd: Beginning new tablet copy session on tablet 39f0a7df4f624eee81c25b41d19d1740 from peer fdfc60a29ea9465c88ee45c3c332caa1 at {username='slave'} at 127.31.161.201:56759: session id = fdfc60a29ea9465c88ee45c3c332caa1-39f0a7df4f624eee81c25b41d19d1740
I20240502 01:54:38.972754 6825 tablet_copy_source_session.cc:205] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:54:38.974422 12524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39f0a7df4f624eee81c25b41d19d1740. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:38.979804 12524 tablet_copy_client.cc:783] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Starting download of 0 data blocks...
I20240502 01:54:38.980285 12524 tablet_copy_client.cc:647] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Starting download of 1 WAL segments...
I20240502 01:54:38.983518 12532 tablet_copy_client.cc:1002] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Injecting latency into file download: 1500
I20240502 01:54:39.021169 9086 consensus_queue.cc:237] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:39.022902 8957 raft_consensus.cc:1261] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc [term 2 FOLLOWER]: Refusing update from remote peer 48866680637c40368e45ba844efb9298: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:54:39.022927 6801 raft_consensus.cc:1261] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd [term 2 FOLLOWER]: Refusing update from remote peer 48866680637c40368e45ba844efb9298: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:54:39.023445 12554 consensus_queue.cc:1035] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:39.023824 12555 consensus_queue.cc:1035] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:54:39.025990 12554 raft_consensus.cc:2932] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:39.026602 6801 raft_consensus.cc:2932] T f6a165cbc26f4defb70866779f83bfdc P 8214b6c4de6d4aa58a22af551eb8dedd [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:39.028560 8957 raft_consensus.cc:2932] T f6a165cbc26f4defb70866779f83bfdc P c39b0639d5fb4d9da09bf1d67d617fcc [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } }
I20240502 01:54:39.030632 6647 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f6a165cbc26f4defb70866779f83bfdc with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:39.032441 6659 catalog_manager.cc:5484] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 reported cstate change: config changed from index 8 to 10, VOTER 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195) evicted. New cstate: current_term: 2 leader_uuid: "48866680637c40368e45ba844efb9298" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:39.036073 7047 tablet_service.cc:1514] Processing DeleteTablet for tablet f6a165cbc26f4defb70866779f83bfdc with delete_type TABLET_DATA_TOMBSTONED (TS 11c097d7d32a4c22a6385a7465876eb3 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:59762
I20240502 01:54:39.036881 12368 tablet_replica.cc:330] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3: stopping tablet replica
I20240502 01:54:39.037103 12368 raft_consensus.cc:2229] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:39.037324 12368 raft_consensus.cc:2258] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:39.038297 12368 ts_tablet_manager.cc:1875] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:39.042363 12368 ts_tablet_manager.cc:1888] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 01:54:39.042459 12368 log.cc:1196] T f6a165cbc26f4defb70866779f83bfdc P 11c097d7d32a4c22a6385a7465876eb3: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/wals/f6a165cbc26f4defb70866779f83bfdc
I20240502 01:54:39.043104 6646 catalog_manager.cc:4830] TS 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083): tablet f6a165cbc26f4defb70866779f83bfdc (table rebalance_test_table_3 [id=76b4b12b25de439c840462959d467432]) successfully deleted
I20240502 01:54:39.802107 12347 tablet_copy_client.cc:523] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:39.805490 12347 tablet_bootstrap.cc:492] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap starting.
I20240502 01:54:39.847213 12347 tablet_bootstrap.cc:492] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:39.847641 12347 tablet_bootstrap.cc:492] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap complete.
I20240502 01:54:39.847910 12347 ts_tablet_manager.cc:1367] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1: Time spent bootstrapping tablet: real 0.042s user 0.032s sys 0.008s
I20240502 01:54:39.848541 12347 raft_consensus.cc:348] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:39.848755 12347 raft_consensus.cc:729] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdfc60a29ea9465c88ee45c3c332caa1, State: Initialized, Role: LEARNER
I20240502 01:54:39.848944 12347 consensus_queue.cc:260] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:39.849843 12347 ts_tablet_manager.cc:1398] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:54:39.850661 6825 tablet_copy_service.cc:342] P 8214b6c4de6d4aa58a22af551eb8dedd: Request end of tablet copy session fdfc60a29ea9465c88ee45c3c332caa1-945224f5cff74edb9447bebc743e5086 received from {username='slave'} at 127.31.161.201:56759
I20240502 01:54:39.850791 6825 tablet_copy_service.cc:433] P 8214b6c4de6d4aa58a22af551eb8dedd: ending tablet copy session fdfc60a29ea9465c88ee45c3c332caa1-945224f5cff74edb9447bebc743e5086 on tablet 945224f5cff74edb9447bebc743e5086 with peer fdfc60a29ea9465c88ee45c3c332caa1
I20240502 01:54:39.927912 12370 tablet_copy_client.cc:523] T 43e6e5b3435a446499106efe7baf92aa P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:39.931617 12370 tablet_bootstrap.cc:492] T 43e6e5b3435a446499106efe7baf92aa P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap starting.
I20240502 01:54:39.963784 12370 tablet_bootstrap.cc:492] T 43e6e5b3435a446499106efe7baf92aa P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:39.964139 12370 tablet_bootstrap.cc:492] T 43e6e5b3435a446499106efe7baf92aa P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap complete.
I20240502 01:54:39.964372 12370 ts_tablet_manager.cc:1367] T 43e6e5b3435a446499106efe7baf92aa P fdfc60a29ea9465c88ee45c3c332caa1: Time spent bootstrapping tablet: real 0.033s user 0.029s sys 0.002s
I20240502 01:54:39.964972 12370 raft_consensus.cc:348] T 43e6e5b3435a446499106efe7baf92aa P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:39.965227 12370 raft_consensus.cc:729] T 43e6e5b3435a446499106efe7baf92aa P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdfc60a29ea9465c88ee45c3c332caa1, State: Initialized, Role: LEARNER
I20240502 01:54:39.965401 12370 consensus_queue.cc:260] T 43e6e5b3435a446499106efe7baf92aa P fdfc60a29ea9465c88ee45c3c332caa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:39.965854 12370 ts_tablet_manager.cc:1398] T 43e6e5b3435a446499106efe7baf92aa P fdfc60a29ea9465c88ee45c3c332caa1: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 01:54:39.966599 7089 tablet_copy_service.cc:342] P 11c097d7d32a4c22a6385a7465876eb3: Request end of tablet copy session fdfc60a29ea9465c88ee45c3c332caa1-43e6e5b3435a446499106efe7baf92aa received from {username='slave'} at 127.31.161.201:57973
I20240502 01:54:39.966748 7089 tablet_copy_service.cc:433] P 11c097d7d32a4c22a6385a7465876eb3: ending tablet copy session fdfc60a29ea9465c88ee45c3c332caa1-43e6e5b3435a446499106efe7baf92aa on tablet 43e6e5b3435a446499106efe7baf92aa with peer fdfc60a29ea9465c88ee45c3c332caa1
I20240502 01:54:40.099089 12433 tablet_copy_client.cc:523] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:40.101655 12433 tablet_bootstrap.cc:492] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap starting.
I20240502 01:54:40.131501 12444 tablet_copy_client.cc:523] T 4b10e180c11c4cd4a308fc7d311840c2 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:40.131776 12433 tablet_bootstrap.cc:492] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:40.132160 12433 tablet_bootstrap.cc:492] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap complete.
I20240502 01:54:40.132442 12433 ts_tablet_manager.cc:1367] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1: Time spent bootstrapping tablet: real 0.031s user 0.020s sys 0.008s
I20240502 01:54:40.133188 12433 raft_consensus.cc:348] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:40.133425 12433 raft_consensus.cc:729] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdfc60a29ea9465c88ee45c3c332caa1, State: Initialized, Role: LEARNER
I20240502 01:54:40.133690 12433 consensus_queue.cc:260] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:40.134132 12433 ts_tablet_manager.cc:1398] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:54:40.134294 12444 tablet_bootstrap.cc:492] T 4b10e180c11c4cd4a308fc7d311840c2 P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap starting.
I20240502 01:54:40.134945 6825 tablet_copy_service.cc:342] P 8214b6c4de6d4aa58a22af551eb8dedd: Request end of tablet copy session fdfc60a29ea9465c88ee45c3c332caa1-a14c3c945b724df5ac3f917cef3186e0 received from {username='slave'} at 127.31.161.201:56759
I20240502 01:54:40.135082 6825 tablet_copy_service.cc:433] P 8214b6c4de6d4aa58a22af551eb8dedd: ending tablet copy session fdfc60a29ea9465c88ee45c3c332caa1-a14c3c945b724df5ac3f917cef3186e0 on tablet a14c3c945b724df5ac3f917cef3186e0 with peer fdfc60a29ea9465c88ee45c3c332caa1
I20240502 01:54:40.168325 12444 tablet_bootstrap.cc:492] T 4b10e180c11c4cd4a308fc7d311840c2 P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:40.168756 12444 tablet_bootstrap.cc:492] T 4b10e180c11c4cd4a308fc7d311840c2 P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap complete.
I20240502 01:54:40.169095 12444 ts_tablet_manager.cc:1367] T 4b10e180c11c4cd4a308fc7d311840c2 P fdfc60a29ea9465c88ee45c3c332caa1: Time spent bootstrapping tablet: real 0.035s user 0.027s sys 0.004s
I20240502 01:54:40.169956 12444 raft_consensus.cc:348] T 4b10e180c11c4cd4a308fc7d311840c2 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:40.170452 12444 raft_consensus.cc:729] T 4b10e180c11c4cd4a308fc7d311840c2 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdfc60a29ea9465c88ee45c3c332caa1, State: Initialized, Role: LEARNER
I20240502 01:54:40.170672 10009 raft_consensus.cc:1203] T 43e6e5b3435a446499106efe7baf92aa P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:40.170763 12444 consensus_queue.cc:260] T 4b10e180c11c4cd4a308fc7d311840c2 P fdfc60a29ea9465c88ee45c3c332caa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:40.171412 12444 ts_tablet_manager.cc:1398] T 4b10e180c11c4cd4a308fc7d311840c2 P fdfc60a29ea9465c88ee45c3c332caa1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:54:40.172106 7089 tablet_copy_service.cc:342] P 11c097d7d32a4c22a6385a7465876eb3: Request end of tablet copy session fdfc60a29ea9465c88ee45c3c332caa1-4b10e180c11c4cd4a308fc7d311840c2 received from {username='slave'} at 127.31.161.201:57973
I20240502 01:54:40.172286 7089 tablet_copy_service.cc:433] P 11c097d7d32a4c22a6385a7465876eb3: ending tablet copy session fdfc60a29ea9465c88ee45c3c332caa1-4b10e180c11c4cd4a308fc7d311840c2 on tablet 4b10e180c11c4cd4a308fc7d311840c2 with peer fdfc60a29ea9465c88ee45c3c332caa1
I20240502 01:54:40.214404 10009 raft_consensus.cc:1203] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:40.329524 10009 raft_consensus.cc:1203] T 4b10e180c11c4cd4a308fc7d311840c2 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:40.483888 12524 tablet_copy_client.cc:523] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:54:40.486611 12524 tablet_bootstrap.cc:492] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap starting.
I20240502 01:54:40.517622 12524 tablet_bootstrap.cc:492] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:54:40.518010 12524 tablet_bootstrap.cc:492] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1: Bootstrap complete.
I20240502 01:54:40.518254 12524 ts_tablet_manager.cc:1367] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1: Time spent bootstrapping tablet: real 0.032s user 0.032s sys 0.000s
I20240502 01:54:40.519030 12524 raft_consensus.cc:348] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:40.519263 12524 raft_consensus.cc:729] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fdfc60a29ea9465c88ee45c3c332caa1, State: Initialized, Role: LEARNER
I20240502 01:54:40.519470 12524 consensus_queue.cc:260] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } }
I20240502 01:54:40.520403 12524 ts_tablet_manager.cc:1398] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:54:40.521283 6825 tablet_copy_service.cc:342] P 8214b6c4de6d4aa58a22af551eb8dedd: Request end of tablet copy session fdfc60a29ea9465c88ee45c3c332caa1-39f0a7df4f624eee81c25b41d19d1740 received from {username='slave'} at 127.31.161.201:56759
I20240502 01:54:40.521425 6825 tablet_copy_service.cc:433] P 8214b6c4de6d4aa58a22af551eb8dedd: ending tablet copy session fdfc60a29ea9465c88ee45c3c332caa1-39f0a7df4f624eee81c25b41d19d1740 on tablet 39f0a7df4f624eee81c25b41d19d1740 with peer fdfc60a29ea9465c88ee45c3c332caa1
I20240502 01:54:40.530232 10009 raft_consensus.cc:1203] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:40.556524 10009 raft_consensus.cc:1203] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 01:54:40.668308 12747 raft_consensus.cc:1050] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3: attempting to promote NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 to VOTER
I20240502 01:54:40.669217 12747 consensus_queue.cc:237] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:40.671846 8957 raft_consensus.cc:1261] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:40.671932 6801 raft_consensus.cc:1261] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:40.672178 10009 raft_consensus.cc:1261] T 43e6e5b3435a446499106efe7baf92aa P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:40.672793 12747 consensus_queue.cc:1035] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:40.673179 12748 consensus_queue.cc:1035] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:40.673550 12517 consensus_queue.cc:1035] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:40.676904 12517 raft_consensus.cc:2932] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:40.677570 6801 raft_consensus.cc:2932] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:40.677915 8957 raft_consensus.cc:2932] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:40.680907 10009 raft_consensus.cc:2932] T 43e6e5b3435a446499106efe7baf92aa P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:40.684006 6659 catalog_manager.cc:5484] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc reported cstate change: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } attrs { replace: true } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:40.694782 7066 consensus_queue.cc:237] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:40.696625 10009 raft_consensus.cc:1261] T 43e6e5b3435a446499106efe7baf92aa P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:40.696738 8957 raft_consensus.cc:1261] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:40.697273 12517 consensus_queue.cc:1035] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:40.697577 12753 consensus_queue.cc:1035] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:40.699890 12753 raft_consensus.cc:2932] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:40.700511 8957 raft_consensus.cc:2932] T 43e6e5b3435a446499106efe7baf92aa P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:40.702970 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 43e6e5b3435a446499106efe7baf92aa with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:40.705369 6657 catalog_manager.cc:5484] T 43e6e5b3435a446499106efe7baf92aa P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index 8 to 9, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:40.706528 10009 raft_consensus.cc:2932] T 43e6e5b3435a446499106efe7baf92aa P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:40.713568 6782 tablet_service.cc:1514] Processing DeleteTablet for tablet 43e6e5b3435a446499106efe7baf92aa with delete_type TABLET_DATA_TOMBSTONED (TS 8214b6c4de6d4aa58a22af551eb8dedd not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:35388
I20240502 01:54:40.714525 12758 tablet_replica.cc:330] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd: stopping tablet replica
I20240502 01:54:40.714938 12758 raft_consensus.cc:2229] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:40.715459 12758 raft_consensus.cc:2258] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:40.716564 12758 ts_tablet_manager.cc:1875] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:40.722092 12758 ts_tablet_manager.cc:1888] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:40.722221 12758 log.cc:1196] T 43e6e5b3435a446499106efe7baf92aa P 8214b6c4de6d4aa58a22af551eb8dedd: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/43e6e5b3435a446499106efe7baf92aa
I20240502 01:54:40.722968 6645 catalog_manager.cc:4830] TS 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): tablet 43e6e5b3435a446499106efe7baf92aa (table rebalance_test_table_7 [id=4d5b7e86d6b24299a377cfaff5756f47]) successfully deleted
I20240502 01:54:40.726385 12760 raft_consensus.cc:1050] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd: attempting to promote NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 to VOTER
I20240502 01:54:40.727212 12760 consensus_queue.cc:237] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:40.730082 7197 raft_consensus.cc:1261] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:40.730105 8692 raft_consensus.cc:1261] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:40.730806 12760 consensus_queue.cc:1035] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:40.731524 12747 raft_consensus.cc:1050] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3: attempting to promote NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 to VOTER
I20240502 01:54:40.731348 12762 consensus_queue.cc:1035] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:40.732383 12747 consensus_queue.cc:237] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:40.732784 10009 raft_consensus.cc:1261] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:40.735280 8823 raft_consensus.cc:1261] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:40.735298 7198 raft_consensus.cc:1261] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:40.735327 12760 consensus_queue.cc:1035] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:40.735841 10009 raft_consensus.cc:1261] T 4b10e180c11c4cd4a308fc7d311840c2 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:40.735955 12753 consensus_queue.cc:1035] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:40.736918 12748 consensus_queue.cc:1035] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:40.737972 12753 consensus_queue.cc:1035] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:40.747357 12760 raft_consensus.cc:2932] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:40.752703 7199 raft_consensus.cc:2932] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:40.752864 8692 raft_consensus.cc:2932] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:40.758201 6657 catalog_manager.cc:5484] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:54:40.764035 12517 raft_consensus.cc:2932] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:40.765105 7198 raft_consensus.cc:2932] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:40.770004 10009 raft_consensus.cc:2932] T 4b10e180c11c4cd4a308fc7d311840c2 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:40.771080 6657 catalog_manager.cc:5484] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:40.771471 10007 raft_consensus.cc:2932] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:40.778543 6801 consensus_queue.cc:237] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:40.780531 10008 raft_consensus.cc:1261] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:40.781391 8692 raft_consensus.cc:1261] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:40.781505 11999 consensus_queue.cc:1035] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:40.782490 12760 consensus_queue.cc:1035] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:40.783787 11999 raft_consensus.cc:2932] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:40.786216 10009 raft_consensus.cc:2932] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:40.786594 8692 raft_consensus.cc:2932] T 945224f5cff74edb9447bebc743e5086 P 3f156b7cfb4f40f481699b257153e5cb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:40.788036 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 945224f5cff74edb9447bebc743e5086 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:40.789610 6657 catalog_manager.cc:5484] T 945224f5cff74edb9447bebc743e5086 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index 8 to 9, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:40.795729 7179 tablet_service.cc:1514] Processing DeleteTablet for tablet 945224f5cff74edb9447bebc743e5086 with delete_type TABLET_DATA_TOMBSTONED (TS fc21155d36d5416fa3650f0268a20b17 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:40840
I20240502 01:54:40.793840 8823 raft_consensus.cc:2932] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:40.798097 7066 consensus_queue.cc:237] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:40.800359 10009 raft_consensus.cc:1261] T 4b10e180c11c4cd4a308fc7d311840c2 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:40.800976 12753 consensus_queue.cc:1035] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:40.805254 12517 raft_consensus.cc:2932] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:40.805997 10009 raft_consensus.cc:2932] T 4b10e180c11c4cd4a308fc7d311840c2 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:40.808485 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4b10e180c11c4cd4a308fc7d311840c2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:40.808952 8822 raft_consensus.cc:1261] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Refusing update from remote peer 11c097d7d32a4c22a6385a7465876eb3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:40.809720 12517 consensus_queue.cc:1035] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:40.810175 6654 catalog_manager.cc:5484] T 4b10e180c11c4cd4a308fc7d311840c2 P 11c097d7d32a4c22a6385a7465876eb3 reported cstate change: config changed from index 8 to 9, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "11c097d7d32a4c22a6385a7465876eb3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:40.811069 8822 raft_consensus.cc:2932] T 4b10e180c11c4cd4a308fc7d311840c2 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER fc21155d36d5416fa3650f0268a20b17 (127.31.161.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:40.812566 12770 tablet_replica.cc:330] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17: stopping tablet replica
I20240502 01:54:40.812790 12770 raft_consensus.cc:2229] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:40.813045 12770 raft_consensus.cc:2258] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:40.814242 12770 ts_tablet_manager.cc:1875] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:40.815232 7178 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b10e180c11c4cd4a308fc7d311840c2 with delete_type TABLET_DATA_TOMBSTONED (TS fc21155d36d5416fa3650f0268a20b17 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:40840
I20240502 01:54:40.821110 12770 ts_tablet_manager.cc:1888] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:40.821318 12770 log.cc:1196] T 945224f5cff74edb9447bebc743e5086 P fc21155d36d5416fa3650f0268a20b17: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/945224f5cff74edb9447bebc743e5086
I20240502 01:54:40.822083 6645 catalog_manager.cc:4830] TS fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873): tablet 945224f5cff74edb9447bebc743e5086 (table rebalance_test_table_9 [id=5b2a4358eba544959466c46759744194]) successfully deleted
I20240502 01:54:40.822125 12770 tablet_replica.cc:330] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17: stopping tablet replica
I20240502 01:54:40.822307 12770 raft_consensus.cc:2229] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:40.822670 12770 raft_consensus.cc:2258] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:40.823603 12770 ts_tablet_manager.cc:1875] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:40.828730 12770 ts_tablet_manager.cc:1888] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:40.828876 12770 log.cc:1196] T 4b10e180c11c4cd4a308fc7d311840c2 P fc21155d36d5416fa3650f0268a20b17: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/4b10e180c11c4cd4a308fc7d311840c2
I20240502 01:54:40.829572 6645 catalog_manager.cc:4830] TS fc21155d36d5416fa3650f0268a20b17 (127.31.161.196:38873): tablet 4b10e180c11c4cd4a308fc7d311840c2 (table rebalance_test_table_8 [id=8abdf44bc9e2414b9e4f80bf4c87c953]) successfully deleted
I20240502 01:54:41.075908 11999 raft_consensus.cc:1050] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd: attempting to promote NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 to VOTER
I20240502 01:54:41.076611 11999 consensus_queue.cc:237] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:41.079005 8822 raft_consensus.cc:1261] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:41.079082 10009 raft_consensus.cc:1261] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:41.079159 6935 raft_consensus.cc:1261] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:41.079716 12760 consensus_queue.cc:1035] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:41.080049 12762 consensus_queue.cc:1035] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:41.080369 12759 consensus_queue.cc:1035] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:41.086503 12759 raft_consensus.cc:2932] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:41.087602 10009 raft_consensus.cc:2932] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:41.090727 8822 raft_consensus.cc:2932] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:41.091547 6936 raft_consensus.cc:2932] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:41.091770 6654 catalog_manager.cc:5484] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:41.102521 6801 consensus_queue.cc:237] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:41.104079 11999 raft_consensus.cc:1050] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd: attempting to promote NON_VOTER fdfc60a29ea9465c88ee45c3c332caa1 to VOTER
I20240502 01:54:41.104818 10009 raft_consensus.cc:1261] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:41.104787 11999 consensus_queue.cc:237] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:41.105371 12760 consensus_queue.cc:1035] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:41.107414 8822 raft_consensus.cc:1261] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:41.108254 12762 consensus_queue.cc:1035] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:41.108891 9086 raft_consensus.cc:1261] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:41.109431 10009 raft_consensus.cc:1261] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 LEARNER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:41.110242 12759 consensus_queue.cc:1035] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:41.110586 12820 consensus_queue.cc:1035] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:41.111189 12762 raft_consensus.cc:2932] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:41.114884 6936 raft_consensus.cc:1261] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:54:41.114784 8822 raft_consensus.cc:2932] T a14c3c945b724df5ac3f917cef3186e0 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:41.115309 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a14c3c945b724df5ac3f917cef3186e0 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:41.116153 12762 consensus_queue.cc:1035] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:54:41.116582 10009 raft_consensus.cc:2932] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:41.116966 6654 catalog_manager.cc:5484] T a14c3c945b724df5ac3f917cef3186e0 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index 8 to 9, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:41.121757 6915 tablet_service.cc:1514] Processing DeleteTablet for tablet a14c3c945b724df5ac3f917cef3186e0 with delete_type TABLET_DATA_TOMBSTONED (TS 137e2183ff57411791e9af58dfab0a3c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:58498
I20240502 01:54:41.122543 12824 tablet_replica.cc:330] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c: stopping tablet replica
I20240502 01:54:41.122960 12824 raft_consensus.cc:2229] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:41.123310 12824 raft_consensus.cc:2258] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:41.124255 12824 ts_tablet_manager.cc:1875] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:41.129354 12820 raft_consensus.cc:2932] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:41.129914 9086 raft_consensus.cc:2932] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:41.131223 12824 ts_tablet_manager.cc:1888] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:41.131392 12824 log.cc:1196] T a14c3c945b724df5ac3f917cef3186e0 P 137e2183ff57411791e9af58dfab0a3c: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/wals/a14c3c945b724df5ac3f917cef3186e0
I20240502 01:54:41.132340 6646 catalog_manager.cc:4830] TS 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315): tablet a14c3c945b724df5ac3f917cef3186e0 (table rebalance_test_table_4 [id=9231a7576c804559bc4a4c54496d5223]) successfully deleted
I20240502 01:54:41.134917 10009 raft_consensus.cc:2932] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:41.135344 6657 catalog_manager.cc:5484] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd reported cstate change: config changed from index 7 to 8, fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:54:41.140810 6801 consensus_queue.cc:237] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:41.144613 9086 raft_consensus.cc:1261] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:41.144891 10009 raft_consensus.cc:1261] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Refusing update from remote peer 8214b6c4de6d4aa58a22af551eb8dedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:54:41.145242 11999 consensus_queue.cc:1035] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:41.145502 12762 consensus_queue.cc:1035] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:54:41.148602 12762 raft_consensus.cc:2932] T 39f0a7df4f624eee81c25b41d19d1740 P 8214b6c4de6d4aa58a22af551eb8dedd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:41.149869 10009 raft_consensus.cc:2932] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:41.150784 9086 raft_consensus.cc:2932] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:41.153265 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 39f0a7df4f624eee81c25b41d19d1740 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:54:41.155709 6654 catalog_manager.cc:5484] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 reported cstate change: config changed from index 8 to 9, VOTER 137e2183ff57411791e9af58dfab0a3c (127.31.161.194) evicted. New cstate: current_term: 1 leader_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } }
I20240502 01:54:41.160233 6915 tablet_service.cc:1514] Processing DeleteTablet for tablet 39f0a7df4f624eee81c25b41d19d1740 with delete_type TABLET_DATA_TOMBSTONED (TS 137e2183ff57411791e9af58dfab0a3c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:58498
I20240502 01:54:41.160596 12824 tablet_replica.cc:330] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c: stopping tablet replica
I20240502 01:54:41.160815 12824 raft_consensus.cc:2229] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:54:41.161027 12824 pending_rounds.cc:62] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c: Trying to abort 1 pending ops.
I20240502 01:54:41.161151 12824 pending_rounds.cc:68] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023062553003393024 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "39f0a7df4f624eee81c25b41d19d1740" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } attrs { replace: true } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } } } }
I20240502 01:54:41.161617 12824 raft_consensus.cc:2894] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240502 01:54:41.161728 12824 raft_consensus.cc:2866] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 01:54:41.161837 12824 raft_consensus.cc:2258] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:54:41.163132 12824 ts_tablet_manager.cc:1875] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:54:41.169348 12824 ts_tablet_manager.cc:1888] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 01:54:41.169531 12824 log.cc:1196] T 39f0a7df4f624eee81c25b41d19d1740 P 137e2183ff57411791e9af58dfab0a3c: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/wals/39f0a7df4f624eee81c25b41d19d1740
I20240502 01:54:41.170346 6646 catalog_manager.cc:4830] TS 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315): tablet 39f0a7df4f624eee81c25b41d19d1740 (table rebalance_test_table_0 [id=145808b6c1a445a19f5a84827cd55658]) successfully deleted
I20240502 01:54:41.325309 7047 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:41.332367 8672 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:41.347121 9067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:41.349063 6915 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:41.356199 6782 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:41.441035 8804 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:41.462060 8937 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:41.490478 7179 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:41.511596 9988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:41.714799 9067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:41.726416 6915 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:41.727124 8672 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:41.727954 7047 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:41.756078 8804 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:41.757565 6782 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:41.822418 8937 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:41.851979 7179 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:41.874814 9988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:42.742727 6915 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:42.744001 7047 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:42.744752 8672 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:42.783224 9067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:42.810474 8804 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:42.824071 6782 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:42.851022 8937 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:42.868253 9988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:42.877763 7179 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:43.088977 7047 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:43.097427 9067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:43.102360 8672 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:43.116300 8937 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:43.117539 6915 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:43.117663 6782 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:43.129539 8804 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:43.136590 7179 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:43.174973 9988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:43.438091 7047 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:43.439194 8672 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:43.448637 9067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:43.451042 6782 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 01:54:43.485520 8804 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:43.501253 8937 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:43.519663 6915 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:43.522918 7179 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:43.549491 9988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:43.891896 6915 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:54:43.892834 9067 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:43.897029 7047 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:54:43.903014 8804 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:43.909235 8672 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:43.957191 7179 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 01:54:43.961393 9988 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:43.962318 8937 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:54:43.966919 6782 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
2a96c230e6a04167a55c5e28a79211e0 | 127.31.161.254:36997 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.31.161.212:43969 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
11c097d7d32a4c22a6385a7465876eb3 | 127.31.161.195:46083 | HEALTHY | <none> | 4 | 0
137e2183ff57411791e9af58dfab0a3c | 127.31.161.194:45315 | HEALTHY | <none> | 5 | 0
3f156b7cfb4f40f481699b257153e5cb | 127.31.161.197:44451 | HEALTHY | <none> | 1 | 0
48866680637c40368e45ba844efb9298 | 127.31.161.200:34863 | HEALTHY | <none> | 1 | 0
8214b6c4de6d4aa58a22af551eb8dedd | 127.31.161.193:45489 | HEALTHY | <none> | 8 | 0
9e47e5c098b848c9abe1559cbcbb6321 | 127.31.161.198:37123 | HEALTHY | <none> | 1 | 0
c39b0639d5fb4d9da09bf1d67d617fcc | 127.31.161.199:35245 | HEALTHY | <none> | 1 | 0
fc21155d36d5416fa3650f0268a20b17 | 127.31.161.196:38873 | HEALTHY | <none> | 9 | 0
fdfc60a29ea9465c88ee45c3c332caa1 | 127.31.161.201:43211 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 9
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
---------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 9 server(s) checked
local_ip_for_outbound_sockets | 127.31.161.193 | experimental | 127.31.161.193:45489
local_ip_for_outbound_sockets | 127.31.161.194 | experimental | 127.31.161.194:45315
local_ip_for_outbound_sockets | 127.31.161.195 | experimental | 127.31.161.195:46083
local_ip_for_outbound_sockets | 127.31.161.196 | experimental | 127.31.161.196:38873
local_ip_for_outbound_sockets | 127.31.161.197 | experimental | 127.31.161.197:44451
local_ip_for_outbound_sockets | 127.31.161.198 | experimental | 127.31.161.198:37123
local_ip_for_outbound_sockets | 127.31.161.199 | experimental | 127.31.161.199:35245
local_ip_for_outbound_sockets | 127.31.161.200 | experimental | 127.31.161.200:34863
local_ip_for_outbound_sockets | 127.31.161.201 | experimental | 127.31.161.201:43211
never_fsync | true | unsafe,advanced | all 9 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 9 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 9 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 9 server(s) checked
server_dump_info_format | pb | hidden | all 9 server(s) checked
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.31.161.193:45489
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.31.161.194:45315
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.31.161.195:46083
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.31.161.196:38873
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.31.161.197:44451
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.31.161.198:37123
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.31.161.199:35245
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.31.161.200:34863
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data/info.pb | hidden | 127.31.161.201:43211
tablet_copy_download_file_inject_latency_ms | 1500 | hidden | all 9 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.31.161.212:43969 | all 9 server(s) checked
time_source | builtin | all 9 server(s) checked
Version Summary
Version | Servers
-----------------+--------------------------
1.18.0-SNAPSHOT | all 10 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 9
Tables | 10
Tablets | 30
Replicas | 90
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 01:54:44.068457 32391 log_verifier.cc:126] Checking tablet 01e98643312b4a2eb80e3bcb3d95674c
I20240502 01:54:44.109627 32391 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 01e98643312b4a2eb80e3bcb3d95674c
I20240502 01:54:44.109750 32391 log_verifier.cc:126] Checking tablet 39f0a7df4f624eee81c25b41d19d1740
I20240502 01:54:44.150985 32391 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 39f0a7df4f624eee81c25b41d19d1740
I20240502 01:54:44.151110 32391 log_verifier.cc:126] Checking tablet 43e6e5b3435a446499106efe7baf92aa
I20240502 01:54:44.223923 32391 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 43e6e5b3435a446499106efe7baf92aa
I20240502 01:54:44.224040 32391 log_verifier.cc:126] Checking tablet 46b6cd134f7e4b2fb56f4df44ed00f15
I20240502 01:54:44.263228 32391 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 46b6cd134f7e4b2fb56f4df44ed00f15
I20240502 01:54:44.263338 32391 log_verifier.cc:126] Checking tablet 482f77b65a254ed3aec717c06698ffa7
I20240502 01:54:44.303658 32391 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 482f77b65a254ed3aec717c06698ffa7
I20240502 01:54:44.303786 32391 log_verifier.cc:126] Checking tablet 4b10e180c11c4cd4a308fc7d311840c2
I20240502 01:54:44.348529 32391 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 4b10e180c11c4cd4a308fc7d311840c2
I20240502 01:54:44.348639 32391 log_verifier.cc:126] Checking tablet 4c1d0974a2fd4a548806b6b6560a89f8
I20240502 01:54:44.388288 32391 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 4c1d0974a2fd4a548806b6b6560a89f8
I20240502 01:54:44.388392 32391 log_verifier.cc:126] Checking tablet 5acc7c2405464a328b2016bb2293f7e8
I20240502 01:54:44.426403 32391 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 5acc7c2405464a328b2016bb2293f7e8
I20240502 01:54:44.426525 32391 log_verifier.cc:126] Checking tablet 640b6474d1834321881094ed0b11b857
I20240502 01:54:44.496217 32391 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 640b6474d1834321881094ed0b11b857
I20240502 01:54:44.496322 32391 log_verifier.cc:126] Checking tablet 70ce4bbee414402aa97c9f3914b9d0c6
I20240502 01:54:44.536274 32391 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 70ce4bbee414402aa97c9f3914b9d0c6
I20240502 01:54:44.536379 32391 log_verifier.cc:126] Checking tablet 76fb5e1f69284140a6c6da264a124b3e
I20240502 01:54:44.574947 32391 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 76fb5e1f69284140a6c6da264a124b3e
I20240502 01:54:44.575078 32391 log_verifier.cc:126] Checking tablet 7d8c4a55e09d41e391b8ba318dfe6896
I20240502 01:54:44.612277 32391 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 7d8c4a55e09d41e391b8ba318dfe6896
I20240502 01:54:44.612382 32391 log_verifier.cc:126] Checking tablet 8960ca8e5d26429589abc4a154311983
I20240502 01:54:44.651088 32391 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 8960ca8e5d26429589abc4a154311983
I20240502 01:54:44.651202 32391 log_verifier.cc:126] Checking tablet 945224f5cff74edb9447bebc743e5086
I20240502 01:54:44.689584 32391 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 945224f5cff74edb9447bebc743e5086
I20240502 01:54:44.689713 32391 log_verifier.cc:126] Checking tablet 98630e8d517a4089b236f32f7616cb00
I20240502 01:54:44.759794 32391 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 98630e8d517a4089b236f32f7616cb00
I20240502 01:54:44.759900 32391 log_verifier.cc:126] Checking tablet a14c3c945b724df5ac3f917cef3186e0
I20240502 01:54:44.802044 32391 log_verifier.cc:177] Verified matching terms for 9 ops in tablet a14c3c945b724df5ac3f917cef3186e0
I20240502 01:54:44.802146 32391 log_verifier.cc:126] Checking tablet ae02d485bc56444e994a56f759fbed1f
I20240502 01:54:44.838083 32391 log_verifier.cc:177] Verified matching terms for 6 ops in tablet ae02d485bc56444e994a56f759fbed1f
I20240502 01:54:44.838182 32391 log_verifier.cc:126] Checking tablet b984c722354c462882428317c411fd78
I20240502 01:54:44.873881 32391 log_verifier.cc:177] Verified matching terms for 6 ops in tablet b984c722354c462882428317c411fd78
I20240502 01:54:44.873976 32391 log_verifier.cc:126] Checking tablet c303b823d03249f89bb6dfaca3934cc9
I20240502 01:54:44.911792 32391 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c303b823d03249f89bb6dfaca3934cc9
I20240502 01:54:44.911891 32391 log_verifier.cc:126] Checking tablet c3397a96d989416fa2bccdc71c7ea9ec
I20240502 01:54:44.955473 32391 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c3397a96d989416fa2bccdc71c7ea9ec
I20240502 01:54:44.955574 32391 log_verifier.cc:126] Checking tablet c86cf318e8e34435bd1dac117d7cc389
I20240502 01:54:45.022635 32391 log_verifier.cc:177] Verified matching terms for 6 ops in tablet c86cf318e8e34435bd1dac117d7cc389
I20240502 01:54:45.022728 32391 log_verifier.cc:126] Checking tablet d1f4c94a105741339ba69e641244ac18
I20240502 01:54:45.059820 32391 log_verifier.cc:177] Verified matching terms for 9 ops in tablet d1f4c94a105741339ba69e641244ac18
I20240502 01:54:45.059916 32391 log_verifier.cc:126] Checking tablet d55da66aad2140d788704b056e476e99
I20240502 01:54:45.097916 32391 log_verifier.cc:177] Verified matching terms for 9 ops in tablet d55da66aad2140d788704b056e476e99
I20240502 01:54:45.098017 32391 log_verifier.cc:126] Checking tablet d58f60c0920e4616850cb246acce1fd7
I20240502 01:54:45.136802 32391 log_verifier.cc:177] Verified matching terms for 11 ops in tablet d58f60c0920e4616850cb246acce1fd7
I20240502 01:54:45.136924 32391 log_verifier.cc:126] Checking tablet dabfafc485c144b995aa4248fb1e897b
I20240502 01:54:45.174525 32391 log_verifier.cc:177] Verified matching terms for 6 ops in tablet dabfafc485c144b995aa4248fb1e897b
I20240502 01:54:45.174640 32391 log_verifier.cc:126] Checking tablet db7a48f131f141f0a4e45036434f8129
I20240502 01:54:45.216718 32391 log_verifier.cc:177] Verified matching terms for 9 ops in tablet db7a48f131f141f0a4e45036434f8129
I20240502 01:54:45.216833 32391 log_verifier.cc:126] Checking tablet dea582c2912241609405695eed436abe
I20240502 01:54:45.280090 32391 log_verifier.cc:177] Verified matching terms for 9 ops in tablet dea582c2912241609405695eed436abe
I20240502 01:54:45.280189 32391 log_verifier.cc:126] Checking tablet ee804f1981f84a118051996fb83056b8
I20240502 01:54:45.318089 32391 log_verifier.cc:177] Verified matching terms for 6 ops in tablet ee804f1981f84a118051996fb83056b8
I20240502 01:54:45.318186 32391 log_verifier.cc:126] Checking tablet f03b37fd2f844f61a39daadc698c9b29
I20240502 01:54:45.356679 32391 log_verifier.cc:177] Verified matching terms for 9 ops in tablet f03b37fd2f844f61a39daadc698c9b29
I20240502 01:54:45.356787 32391 log_verifier.cc:126] Checking tablet f6a165cbc26f4defb70866779f83bfdc
I20240502 01:54:45.395931 32391 log_verifier.cc:177] Verified matching terms for 10 ops in tablet f6a165cbc26f4defb70866779f83bfdc
I20240502 01:54:45.396746 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 6720
I20240502 01:54:45.403704 6845 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:45.633069 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 6720
W20240502 01:54:45.673612 8757 connection.cc:535] server connection from 127.31.161.193:58213 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:45.673913 8758 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.161.193:45489 (error 108)
I20240502 01:54:45.675215 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 6854
W20240502 01:54:45.676514 8758 consensus_peers.cc:485] T d58f60c0920e4616850cb246acce1fd7 P 9e47e5c098b848c9abe1559cbcbb6321 -> Peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): Couldn't send request to peer 8214b6c4de6d4aa58a22af551eb8dedd. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.193:45489: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:45.687681 6978 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:45.913317 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 6854
W20240502 01:54:45.947247 8892 connection.cc:535] client connection to 127.31.161.194:45315 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:45.947547 8892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:45.948240 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 6986
W20240502 01:54:45.949668 8892 consensus_peers.cc:485] T 4c1d0974a2fd4a548806b6b6560a89f8 P c39b0639d5fb4d9da09bf1d67d617fcc -> Peer 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315): Couldn't send request to peer 137e2183ff57411791e9af58dfab0a3c. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.194:45315: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:45.954530 7107 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:54:46.145375 9021 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.161.193:45489: connect: Connection refused (error 111)
W20240502 01:54:46.146840 9021 consensus_peers.cc:485] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 -> Peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): Couldn't send request to peer 8214b6c4de6d4aa58a22af551eb8dedd. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.193:45489: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 01:54:46.150502 7134 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.161.194:45315: connect: Connection refused (error 111)
W20240502 01:54:46.151914 7134 consensus_peers.cc:485] T ae02d485bc56444e994a56f759fbed1f P fc21155d36d5416fa3650f0268a20b17 -> Peer 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315): Couldn't send request to peer 137e2183ff57411791e9af58dfab0a3c. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.194:45315: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:46.171270 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 6986
W20240502 01:54:46.206523 7135 connection.cc:535] server connection from 127.31.161.195:47719 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:46.207376 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 7117
I20240502 01:54:46.213768 7237 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:46.371645 12633 raft_consensus.cc:482] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8214b6c4de6d4aa58a22af551eb8dedd)
I20240502 01:54:46.371876 12633 raft_consensus.cc:504] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:46.372797 12633 leader_election.cc:288] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201:43211)
W20240502 01:54:46.374490 9021 leader_election.cc:334] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): Network error: Client connection negotiation failed: client connection to 127.31.161.193:45489: connect: Connection refused (error 111)
I20240502 01:54:46.379779 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8960ca8e5d26429589abc4a154311983" candidate_uuid: "48866680637c40368e45ba844efb9298" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" is_pre_election: true
I20240502 01:54:46.380465 9022 leader_election.cc:302] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 48866680637c40368e45ba844efb9298; no voters: 8214b6c4de6d4aa58a22af551eb8dedd, fdfc60a29ea9465c88ee45c3c332caa1
I20240502 01:54:46.380828 12633 raft_consensus.cc:2726] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 01:54:46.417781 8626 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.161.194:45315: connect: Connection refused (error 111)
W20240502 01:54:46.419250 8626 consensus_peers.cc:485] T 70ce4bbee414402aa97c9f3914b9d0c6 P 3f156b7cfb4f40f481699b257153e5cb -> Peer 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315): Couldn't send request to peer 137e2183ff57411791e9af58dfab0a3c. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.194:45315: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:46.433779 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 7117
I20240502 01:54:46.467808 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 8609
I20240502 01:54:46.474141 8730 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:46.484505 13139 raft_consensus.cc:482] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8214b6c4de6d4aa58a22af551eb8dedd)
I20240502 01:54:46.484705 13139 raft_consensus.cc:504] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:46.485633 13139 leader_election.cc:288] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198:37123)
W20240502 01:54:46.488026 8891 leader_election.cc:334] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): Network error: Client connection negotiation failed: client connection to 127.31.161.193:45489: connect: Connection refused (error 111)
I20240502 01:54:46.491998 8822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1f4c94a105741339ba69e641244ac18" candidate_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "9e47e5c098b848c9abe1559cbcbb6321" is_pre_election: true
I20240502 01:54:46.492750 8892 leader_election.cc:302] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c39b0639d5fb4d9da09bf1d67d617fcc; no voters: 8214b6c4de6d4aa58a22af551eb8dedd, 9e47e5c098b848c9abe1559cbcbb6321
I20240502 01:54:46.493108 13139 raft_consensus.cc:2726] T d1f4c94a105741339ba69e641244ac18 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:46.621963 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 8609
I20240502 01:54:46.629283 13131 raft_consensus.cc:482] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8214b6c4de6d4aa58a22af551eb8dedd)
I20240502 01:54:46.629487 13131 raft_consensus.cc:504] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:46.630404 13131 leader_election.cc:288] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199:35245)
I20240502 01:54:46.650171 8957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f03b37fd2f844f61a39daadc698c9b29" candidate_uuid: "9e47e5c098b848c9abe1559cbcbb6321" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" is_pre_election: true
W20240502 01:54:46.650949 8758 leader_election.cc:334] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): Network error: Client connection negotiation failed: client connection to 127.31.161.193:45489: connect: Connection refused (error 111)
I20240502 01:54:46.651189 8759 leader_election.cc:302] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e47e5c098b848c9abe1559cbcbb6321; no voters: 8214b6c4de6d4aa58a22af551eb8dedd, c39b0639d5fb4d9da09bf1d67d617fcc
I20240502 01:54:46.651486 13131 raft_consensus.cc:2726] T f03b37fd2f844f61a39daadc698c9b29 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:46.654541 13140 raft_consensus.cc:482] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 137e2183ff57411791e9af58dfab0a3c)
I20240502 01:54:46.654800 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 8741
I20240502 01:54:46.654759 13140 raft_consensus.cc:504] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:46.655728 13140 leader_election.cc:288] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315), 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083)
W20240502 01:54:46.657204 8892 leader_election.cc:334] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315): Network error: Client connection negotiation failed: client connection to 127.31.161.194:45315: connect: Connection refused (error 111)
W20240502 01:54:46.657414 8892 leader_election.cc:334] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083): Network error: Client connection negotiation failed: client connection to 127.31.161.195:46083: connect: Connection refused (error 111)
I20240502 01:54:46.657516 8892 leader_election.cc:302] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c39b0639d5fb4d9da09bf1d67d617fcc; no voters: 11c097d7d32a4c22a6385a7465876eb3, 137e2183ff57411791e9af58dfab0a3c
I20240502 01:54:46.657778 13140 raft_consensus.cc:2726] T 98630e8d517a4089b236f32f7616cb00 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:46.660693 8863 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:46.714608 13140 raft_consensus.cc:482] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8214b6c4de6d4aa58a22af551eb8dedd)
I20240502 01:54:46.714846 13140 raft_consensus.cc:504] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } }
I20240502 01:54:46.715514 13140 leader_election.cc:288] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083), 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489)
W20240502 01:54:46.716977 8891 leader_election.cc:334] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): Network error: Client connection negotiation failed: client connection to 127.31.161.193:45489: connect: Connection refused (error 111)
W20240502 01:54:46.717175 8892 leader_election.cc:334] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 11c097d7d32a4c22a6385a7465876eb3 (127.31.161.195:46083): Network error: Client connection negotiation failed: client connection to 127.31.161.195:46083: connect: Connection refused (error 111)
I20240502 01:54:46.717356 8892 leader_election.cc:302] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c39b0639d5fb4d9da09bf1d67d617fcc; no voters: 11c097d7d32a4c22a6385a7465876eb3, 8214b6c4de6d4aa58a22af551eb8dedd
I20240502 01:54:46.717607 13140 raft_consensus.cc:2726] T c303b823d03249f89bb6dfaca3934cc9 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:46.723446 13159 raft_consensus.cc:482] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8214b6c4de6d4aa58a22af551eb8dedd)
I20240502 01:54:46.723652 13159 raft_consensus.cc:504] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:46.725302 13159 leader_election.cc:288] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), 48866680637c40368e45ba844efb9298 (127.31.161.200:34863)
W20240502 01:54:46.738327 9927 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.161.193:45489: connect: Connection refused (error 111)
W20240502 01:54:46.743434 9927 leader_election.cc:334] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): Network error: Client connection negotiation failed: client connection to 127.31.161.193:45489: connect: Connection refused (error 111)
I20240502 01:54:46.747191 9086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f0a7df4f624eee81c25b41d19d1740" candidate_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "48866680637c40368e45ba844efb9298" is_pre_election: true
I20240502 01:54:46.747953 9929 leader_election.cc:302] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfc60a29ea9465c88ee45c3c332caa1; no voters: 48866680637c40368e45ba844efb9298, 8214b6c4de6d4aa58a22af551eb8dedd
I20240502 01:54:46.748332 13159 raft_consensus.cc:2726] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:46.787184 13159 raft_consensus.cc:482] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8214b6c4de6d4aa58a22af551eb8dedd)
I20240502 01:54:46.787410 13159 raft_consensus.cc:504] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:46.788220 13159 leader_election.cc:288] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198:37123)
W20240502 01:54:46.789512 9927 leader_election.cc:334] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): Network error: Client connection negotiation failed: client connection to 127.31.161.193:45489: connect: Connection refused (error 111)
I20240502 01:54:46.806835 13164 raft_consensus.cc:482] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8214b6c4de6d4aa58a22af551eb8dedd)
I20240502 01:54:46.807065 13164 raft_consensus.cc:504] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:46.807924 13164 leader_election.cc:288] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201:43211)
I20240502 01:54:46.808630 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f0a7df4f624eee81c25b41d19d1740" candidate_uuid: "48866680637c40368e45ba844efb9298" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" is_pre_election: true
I20240502 01:54:46.808889 10009 raft_consensus.cc:2454] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48866680637c40368e45ba844efb9298 in term 1.
I20240502 01:54:46.809314 9022 leader_election.cc:302] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48866680637c40368e45ba844efb9298, fdfc60a29ea9465c88ee45c3c332caa1; no voters:
W20240502 01:54:46.809530 9021 leader_election.cc:334] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): Network error: Client connection negotiation failed: client connection to 127.31.161.193:45489: connect: Connection refused (error 111)
I20240502 01:54:46.809624 13164 raft_consensus.cc:2781] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:46.809791 13164 raft_consensus.cc:482] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8214b6c4de6d4aa58a22af551eb8dedd)
I20240502 01:54:46.809890 13164 raft_consensus.cc:3037] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:46.812927 13164 raft_consensus.cc:504] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:46.813716 13164 leader_election.cc:288] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [CANDIDATE]: Term 2 election: Requested vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201:43211)
I20240502 01:54:46.814381 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39f0a7df4f624eee81c25b41d19d1740" candidate_uuid: "48866680637c40368e45ba844efb9298" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "fdfc60a29ea9465c88ee45c3c332caa1"
I20240502 01:54:46.814610 10009 raft_consensus.cc:3037] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:46.816526 10009 raft_consensus.cc:2454] T 39f0a7df4f624eee81c25b41d19d1740 P fdfc60a29ea9465c88ee45c3c332caa1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48866680637c40368e45ba844efb9298 in term 2.
I20240502 01:54:46.816979 9022 leader_election.cc:302] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48866680637c40368e45ba844efb9298, fdfc60a29ea9465c88ee45c3c332caa1; no voters:
W20240502 01:54:46.817189 9021 leader_election.cc:334] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): Network error: Client connection negotiation failed: client connection to 127.31.161.193:45489: connect: Connection refused (error 111)
I20240502 01:54:46.817307 13164 raft_consensus.cc:2781] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:54:46.817466 13164 raft_consensus.cc:686] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [term 2 LEADER]: Becoming Leader. State: Replica: 48866680637c40368e45ba844efb9298, State: Running, Role: LEADER
I20240502 01:54:46.817847 13164 consensus_queue.cc:237] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:46.819219 13159 raft_consensus.cc:482] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 137e2183ff57411791e9af58dfab0a3c)
I20240502 01:54:46.819437 13159 raft_consensus.cc:504] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:46.820142 13159 leader_election.cc:288] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315), 48866680637c40368e45ba844efb9298 (127.31.161.200:34863)
I20240502 01:54:46.820880 9086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea582c2912241609405695eed436abe" candidate_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "48866680637c40368e45ba844efb9298" is_pre_election: true
I20240502 01:54:46.821310 6651 catalog_manager.cc:5484] T 39f0a7df4f624eee81c25b41d19d1740 P 48866680637c40368e45ba844efb9298 reported cstate change: term changed from 1 to 2, leader changed from 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193) to 48866680637c40368e45ba844efb9298 (127.31.161.200). New cstate: current_term: 2 leader_uuid: "48866680637c40368e45ba844efb9298" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 01:54:46.822301 9928 leader_election.cc:334] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315): Network error: Client connection negotiation failed: client connection to 127.31.161.194:45315: connect: Connection refused (error 111)
I20240502 01:54:46.822428 9928 leader_election.cc:302] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfc60a29ea9465c88ee45c3c332caa1; no voters: 137e2183ff57411791e9af58dfab0a3c, 48866680637c40368e45ba844efb9298
I20240502 01:54:46.822695 13159 raft_consensus.cc:2726] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:46.830158 13131 raft_consensus.cc:482] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8214b6c4de6d4aa58a22af551eb8dedd)
I20240502 01:54:46.830343 13131 raft_consensus.cc:504] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:46.831157 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 8741
I20240502 01:54:46.831161 13131 leader_election.cc:288] T d1f4c94a105741339ba69e641244ac18 P 9e47e5c098b848c9abe1559cbcbb6321 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199:35245)
I20240502 01:54:46.854229 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 8873
W20240502 01:54:46.855031 13160 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 01:54:46.788465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.161.198:37123 (local address 127.31.161.201:57919)
0502 01:54:46.788761 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0502 01:54:46.788778 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0502 01:54:46.788905 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 01:54:46.831668 (+ 42763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0502 01:54:46.831675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0502 01:54:46.831709 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0502 01:54:46.832261 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0502 01:54:46.832272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0502 01:54:46.854131 (+ 21859us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.161.198:37123: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":197}
W20240502 01:54:46.856278 9928 leader_election.cc:334] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198:37123): Network error: Client connection negotiation failed: client connection to 127.31.161.198:37123: connect: Connection refused (error 111)
I20240502 01:54:46.856431 9928 leader_election.cc:302] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfc60a29ea9465c88ee45c3c332caa1; no voters: 8214b6c4de6d4aa58a22af551eb8dedd, 9e47e5c098b848c9abe1559cbcbb6321
I20240502 01:54:46.856717 13159 raft_consensus.cc:2726] T a14c3c945b724df5ac3f917cef3186e0 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:46.860144 8996 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:46.863219 13159 raft_consensus.cc:482] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8214b6c4de6d4aa58a22af551eb8dedd)
I20240502 01:54:46.863413 13159 raft_consensus.cc:504] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "3f156b7cfb4f40f481699b257153e5cb" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 44451 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:46.864229 13159 leader_election.cc:288] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197:44451)
W20240502 01:54:46.865386 9927 leader_election.cc:334] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): Network error: Client connection negotiation failed: client connection to 127.31.161.193:45489: connect: Connection refused (error 111)
W20240502 01:54:46.865602 9928 leader_election.cc:334] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3f156b7cfb4f40f481699b257153e5cb (127.31.161.197:44451): Network error: Client connection negotiation failed: client connection to 127.31.161.197:44451: connect: Connection refused (error 111)
I20240502 01:54:46.865712 9928 leader_election.cc:302] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfc60a29ea9465c88ee45c3c332caa1; no voters: 3f156b7cfb4f40f481699b257153e5cb, 8214b6c4de6d4aa58a22af551eb8dedd
I20240502 01:54:46.865964 13159 raft_consensus.cc:2726] T 945224f5cff74edb9447bebc743e5086 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:54:47.009346 13159 raft_consensus.cc:482] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 137e2183ff57411791e9af58dfab0a3c)
I20240502 01:54:47.009578 13159 raft_consensus.cc:504] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:47.010332 13159 leader_election.cc:288] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315), 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198:37123)
I20240502 01:54:47.010946 13139 raft_consensus.cc:482] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8214b6c4de6d4aa58a22af551eb8dedd)
I20240502 01:54:47.011137 13139 raft_consensus.cc:504] T f03b37fd2f844f61a39daadc698c9b29 P c39b0639d5fb4d9da09bf1d67d617fcc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } }
I20240502 01:54:47.011502 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 8873
W20240502 01:54:47.013604 9928 leader_election.cc:334] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315): Network error: Client connection negotiation failed: client connection to 127.31.161.194:45315: connect: Connection refused (error 111)
W20240502 01:54:47.013772 9928 leader_election.cc:334] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9e47e5c098b848c9abe1559cbcbb6321 (127.31.161.198:37123): Network error: Client connection negotiation failed: client connection to 127.31.161.198:37123: connect: Connection refused (error 111)
I20240502 01:54:47.013878 9928 leader_election.cc:302] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfc60a29ea9465c88ee45c3c332caa1; no voters: 137e2183ff57411791e9af58dfab0a3c, 9e47e5c098b848c9abe1559cbcbb6321
I20240502 01:54:47.014142 13159 raft_consensus.cc:2726] T 01e98643312b4a2eb80e3bcb3d95674c P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 01:54:47.033881 9023 connection.cc:535] server connection from 127.31.161.199:40991 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:47.034039 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 9005
I20240502 01:54:47.039461 9126 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:47.132894 13159 raft_consensus.cc:482] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8214b6c4de6d4aa58a22af551eb8dedd)
I20240502 01:54:47.133137 13159 raft_consensus.cc:504] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:47.133888 13159 leader_election.cc:288] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), 48866680637c40368e45ba844efb9298 (127.31.161.200:34863)
W20240502 01:54:47.135242 9927 leader_election.cc:334] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): Network error: Client connection negotiation failed: client connection to 127.31.161.193:45489: connect: Connection refused (error 111)
I20240502 01:54:47.159979 13159 raft_consensus.cc:482] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 137e2183ff57411791e9af58dfab0a3c)
I20240502 01:54:47.160241 13159 raft_consensus.cc:504] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:47.160938 13159 leader_election.cc:288] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315), 48866680637c40368e45ba844efb9298 (127.31.161.200:34863)
W20240502 01:54:47.162293 9928 leader_election.cc:334] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315): Network error: Client connection negotiation failed: client connection to 127.31.161.194:45315: connect: Connection refused (error 111)
I20240502 01:54:47.189488 9086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8960ca8e5d26429589abc4a154311983" candidate_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "48866680637c40368e45ba844efb9298" is_pre_election: true
I20240502 01:54:47.189862 9086 raft_consensus.cc:2454] T 8960ca8e5d26429589abc4a154311983 P 48866680637c40368e45ba844efb9298 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfc60a29ea9465c88ee45c3c332caa1 in term 2.
I20240502 01:54:47.190366 13164 raft_consensus.cc:482] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 137e2183ff57411791e9af58dfab0a3c)
I20240502 01:54:47.190551 13163 raft_consensus.cc:482] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 137e2183ff57411791e9af58dfab0a3c)
I20240502 01:54:47.190699 13163 raft_consensus.cc:504] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:47.191239 13164 raft_consensus.cc:504] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:47.191466 9085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fb5e1f69284140a6c6da264a124b3e" candidate_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "48866680637c40368e45ba844efb9298" is_pre_election: true
I20240502 01:54:47.191668 9085 raft_consensus.cc:2454] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfc60a29ea9465c88ee45c3c332caa1 in term 1.
I20240502 01:54:47.191664 13163 leader_election.cc:288] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315), fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201:43211)
I20240502 01:54:47.191923 13164 leader_election.cc:288] T 76fb5e1f69284140a6c6da264a124b3e P 48866680637c40368e45ba844efb9298 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315), fdfc60a29ea9465c88ee45c3c332caa1 (127.31.161.201:43211)
I20240502 01:54:47.192324 9929 leader_election.cc:302] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48866680637c40368e45ba844efb9298, fdfc60a29ea9465c88ee45c3c332caa1; no voters: 8214b6c4de6d4aa58a22af551eb8dedd
I20240502 01:54:47.192785 13159 raft_consensus.cc:2781] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 01:54:47.192874 9929 leader_election.cc:302] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 48866680637c40368e45ba844efb9298, fdfc60a29ea9465c88ee45c3c332caa1; no voters: 137e2183ff57411791e9af58dfab0a3c
I20240502 01:54:47.192898 13159 raft_consensus.cc:482] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8214b6c4de6d4aa58a22af551eb8dedd)
I20240502 01:54:47.193522 13159 raft_consensus.cc:3037] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [term 2 FOLLOWER]: Advancing to term 3
W20240502 01:54:47.193572 9022 consensus_peers.cc:485] T f6a165cbc26f4defb70866779f83bfdc P 48866680637c40368e45ba844efb9298 -> Peer c39b0639d5fb4d9da09bf1d67d617fcc (127.31.161.199:35245): Couldn't send request to peer c39b0639d5fb4d9da09bf1d67d617fcc. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.199:35245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:54:47.194126 10009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dea582c2912241609405695eed436abe" candidate_uuid: "48866680637c40368e45ba844efb9298" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" is_pre_election: true
I20240502 01:54:47.194324 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 9005
I20240502 01:54:47.194365 10009 raft_consensus.cc:2454] T dea582c2912241609405695eed436abe P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48866680637c40368e45ba844efb9298 in term 1.
W20240502 01:54:47.194379 9022 leader_election.cc:334] T dea582c2912241609405695eed436abe P 48866680637c40368e45ba844efb9298 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315): Network error: Client connection negotiation failed: client connection to 127.31.161.194:45315: connect: Connection refused (error 111)
I20240502 01:54:47.194636 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76fb5e1f69284140a6c6da264a124b3e" candidate_uuid: "48866680637c40368e45ba844efb9298" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" is_pre_election: true
I20240502 01:54:47.194774 10008 raft_consensus.cc:2454] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48866680637c40368e45ba844efb9298 in term 1.
I20240502 01:54:47.197463 13176 raft_consensus.cc:2781] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 01:54:47.197567 13176 raft_consensus.cc:482] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 137e2183ff57411791e9af58dfab0a3c)
I20240502 01:54:47.197706 13176 raft_consensus.cc:3037] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:54:47.198150 13159 raft_consensus.cc:504] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "8214b6c4de6d4aa58a22af551eb8dedd" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45489 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:47.198931 13159 leader_election.cc:288] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 3 election: Requested vote from peers 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489), 48866680637c40368e45ba844efb9298 (127.31.161.200:34863)
I20240502 01:54:47.200131 13176 raft_consensus.cc:504] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "137e2183ff57411791e9af58dfab0a3c" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 45315 } } peers { permanent_uuid: "48866680637c40368e45ba844efb9298" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 34863 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
W20240502 01:54:47.200335 9927 leader_election.cc:334] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8214b6c4de6d4aa58a22af551eb8dedd (127.31.161.193:45489): Network error: Client connection negotiation failed: client connection to 127.31.161.193:45489: connect: Connection refused (error 111)
I20240502 01:54:47.200793 13176 leader_election.cc:288] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 election: Requested vote from peers 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315), 48866680637c40368e45ba844efb9298 (127.31.161.200:34863)
W20240502 01:54:47.201712 9928 leader_election.cc:334] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 137e2183ff57411791e9af58dfab0a3c (127.31.161.194:45315): Network error: Client connection negotiation failed: client connection to 127.31.161.194:45315: connect: Connection refused (error 111)
W20240502 01:54:47.219309 9928 connection.cc:535] server connection from 127.31.161.200:59035 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 01:54:47.219309 9929 connection.cc:535] client connection to 127.31.161.200:34863 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:54:47.219892 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 9811
W20240502 01:54:47.221015 9929 leader_election.cc:334] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 48866680637c40368e45ba844efb9298 (127.31.161.200:34863): Network error: Client connection negotiation failed: client connection to 127.31.161.200:34863: connect: Connection refused (error 111)
I20240502 01:54:47.221189 9929 leader_election.cc:302] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfc60a29ea9465c88ee45c3c332caa1; no voters: 48866680637c40368e45ba844efb9298, 8214b6c4de6d4aa58a22af551eb8dedd
I20240502 01:54:47.221539 13176 raft_consensus.cc:2726] T 8960ca8e5d26429589abc4a154311983 P fdfc60a29ea9465c88ee45c3c332caa1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20240502 01:54:47.221596 9929 leader_election.cc:334] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 48866680637c40368e45ba844efb9298 (127.31.161.200:34863): Network error: Client connection negotiation failed: client connection to 127.31.161.200:34863: connect: Connection refused (error 111)
I20240502 01:54:47.221727 9929 leader_election.cc:302] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fdfc60a29ea9465c88ee45c3c332caa1; no voters: 137e2183ff57411791e9af58dfab0a3c, 48866680637c40368e45ba844efb9298
I20240502 01:54:47.221973 13176 raft_consensus.cc:2726] T 76fb5e1f69284140a6c6da264a124b3e P fdfc60a29ea9465c88ee45c3c332caa1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 01:54:47.226255 10049 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:47.353629 13176 raft_consensus.cc:482] T 5acc7c2405464a328b2016bb2293f7e8 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fc21155d36d5416fa3650f0268a20b17)
I20240502 01:54:47.353632 13159 raft_consensus.cc:482] T 43e6e5b3435a446499106efe7baf92aa P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 11c097d7d32a4c22a6385a7465876eb3)
I20240502 01:54:47.354044 13159 raft_consensus.cc:504] T 43e6e5b3435a446499106efe7baf92aa P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "11c097d7d32a4c22a6385a7465876eb3" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 46083 } } peers { permanent_uuid: "c39b0639d5fb4d9da09bf1d67d617fcc" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 35245 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:47.354087 13176 raft_consensus.cc:504] T 5acc7c2405464a328b2016bb2293f7e8 P fdfc60a29ea9465c88ee45c3c332caa1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "fc21155d36d5416fa3650f0268a20b17" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 38873 } } peers { permanent_uuid: "9e47e5c098b848c9abe1559cbcbb6321" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 37123 } attrs { promote: false } } peers { permanent_uuid: "fdfc60a29ea9465c88ee45c3c332caa1" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 43211 } attrs { promote: false } }
I20240502 01:54:47.354604 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 9811
I20240502 01:54:47.375185 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 6629
I20240502 01:54:47.376361 6689 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:47.581452 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 6629
2024-05-02T01:54:47Z chronyd exiting
[ OK ] TserverAddedDuringRebalancingTest.TserverStarts/1 (46446 ms)
[----------] 1 test from TserverAddedDuringRebalancingTest (46446 ms total)
[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/5
I20240502 01:54:47.724633 32391 test_util.cc:268] Using random seed: -481348041
I20240502 01:54:47.726464 32391 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:54:47.726539 32391 ts_itest-base.cc:113] --------------
I20240502 01:54:47.726576 32391 ts_itest-base.cc:114] 4 tablet servers
I20240502 01:54:47.726634 32391 ts_itest-base.cc:115] 1 replicas per TS
I20240502 01:54:47.726666 32391 ts_itest-base.cc:116] --------------
2024-05-02T01:54:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:54:47Z Disabled control of system clock
I20240502 01:54:47.775619 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.161.254:34193
--webserver_interface=127.31.161.254
--webserver_port=0
--builtin_ntp_servers=127.31.161.212:43169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.161.254:34193
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240502 01:54:48.108757 13187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:48.109062 13187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:48.109189 13187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:48.116667 13187 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:48.116789 13187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:54:48.116842 13187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:48.116896 13187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:54:48.116943 13187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:54:48.126304 13187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:43169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.31.161.254:34193
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.161.254:34193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.31.161.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:48.127596 13187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:48.129280 13187 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:48.139428 13192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:48.140017 13193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:48.140707 13195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:48.140964 13187 server_base.cc:1034] running on GCE node
I20240502 01:54:48.141742 13187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:48.143092 13187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:48.144321 13187 hybrid_clock.cc:648] HybridClock initialized: now 1714614888144261 us; error 55 us; skew 500 ppm
I20240502 01:54:48.144645 13187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:48.146894 13187 webserver.cc:460] Webserver started at http://127.31.161.254:44073/ using document root <none> and password file <none>
I20240502 01:54:48.147631 13187 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:48.147748 13187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:48.148039 13187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:48.150637 13187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2e9fbea5b73148cbac695089ef8dc217"
format_stamp: "Formatted at 2024-05-02 01:54:48 on dist-test-slave-06l8"
I20240502 01:54:48.151391 13187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2e9fbea5b73148cbac695089ef8dc217"
format_stamp: "Formatted at 2024-05-02 01:54:48 on dist-test-slave-06l8"
I20240502 01:54:48.156353 13187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 01:54:48.159561 13201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:48.160821 13187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:48.160982 13187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2e9fbea5b73148cbac695089ef8dc217"
format_stamp: "Formatted at 2024-05-02 01:54:48 on dist-test-slave-06l8"
I20240502 01:54:48.161219 13187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:48.196820 13187 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:48.197782 13187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:48.198081 13187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:48.215771 13187 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.254:34193
I20240502 01:54:48.215780 13253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.254:34193 every 8 connection(s)
I20240502 01:54:48.217142 13187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:54:48.220808 13254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:48.223683 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 13187
I20240502 01:54:48.223893 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:54:48.230221 13254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217: Bootstrap starting.
I20240502 01:54:48.233357 13254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:48.234350 13254 log.cc:824] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:48.236909 13254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217: No bootstrap required, opened a new log
I20240502 01:54:48.243656 13254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9fbea5b73148cbac695089ef8dc217" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 34193 } }
I20240502 01:54:48.243988 13254 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:48.244103 13254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e9fbea5b73148cbac695089ef8dc217, State: Initialized, Role: FOLLOWER
I20240502 01:54:48.244721 13254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9fbea5b73148cbac695089ef8dc217" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 34193 } }
I20240502 01:54:48.244935 13254 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:48.245059 13254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:48.245239 13254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:48.247278 13254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9fbea5b73148cbac695089ef8dc217" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 34193 } }
I20240502 01:54:48.247759 13254 leader_election.cc:302] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e9fbea5b73148cbac695089ef8dc217; no voters:
I20240502 01:54:48.248441 13254 leader_election.cc:288] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:54:48.248571 13257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:48.250726 13254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:54:48.255427 13257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217 [term 1 LEADER]: Becoming Leader. State: Replica: 2e9fbea5b73148cbac695089ef8dc217, State: Running, Role: LEADER
I20240502 01:54:48.256103 13257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9fbea5b73148cbac695089ef8dc217" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 34193 } }
I20240502 01:54:48.266049 13258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e9fbea5b73148cbac695089ef8dc217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9fbea5b73148cbac695089ef8dc217" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 34193 } } }
I20240502 01:54:48.266435 13258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217 [sys.catalog]: This master's current role is: LEADER
I20240502 01:54:48.266234 13259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e9fbea5b73148cbac695089ef8dc217. Latest consensus state: current_term: 1 leader_uuid: "2e9fbea5b73148cbac695089ef8dc217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e9fbea5b73148cbac695089ef8dc217" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 34193 } } }
I20240502 01:54:48.266577 13259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217 [sys.catalog]: This master's current role is: LEADER
I20240502 01:54:48.269641 13270 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:54:48.275527 13270 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:54:48.285880 13270 catalog_manager.cc:1347] Generated new cluster ID: 0a726f38872049ee80aa5766143c8f7d
I20240502 01:54:48.285991 13270 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:54:48.302377 13270 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:54:48.303728 13270 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:54:48.313035 13270 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2e9fbea5b73148cbac695089ef8dc217: Generated new TSK 0
I20240502 01:54:48.314054 13270 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:54:48.319638 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.193:0
--local_ip_for_outbound_sockets=127.31.161.193
--webserver_interface=127.31.161.193
--webserver_port=0
--tserver_master_addrs=127.31.161.254:34193
--builtin_ntp_servers=127.31.161.212:43169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:54:48.653405 13279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:48.653692 13279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:48.653824 13279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:48.661283 13279 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:48.661424 13279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:48.661574 13279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.193
I20240502 01:54:48.670838 13279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:43169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.31.161.193
--webserver_port=0
--tserver_master_addrs=127.31.161.254:34193
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.193
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:48.672143 13279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:48.674033 13279 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:48.685359 13284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:48.685997 13287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:48.686720 13285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:48.690713 13279 server_base.cc:1034] running on GCE node
I20240502 01:54:48.691349 13279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:48.692315 13279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:48.693540 13279 hybrid_clock.cc:648] HybridClock initialized: now 1714614888693483 us; error 45 us; skew 500 ppm
I20240502 01:54:48.693861 13279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:48.696125 13279 webserver.cc:460] Webserver started at http://127.31.161.193:38563/ using document root <none> and password file <none>
I20240502 01:54:48.696861 13279 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:48.696983 13279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:48.697322 13279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:48.700007 13279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b3ea4e38a26a477aa838e668484b2338"
format_stamp: "Formatted at 2024-05-02 01:54:48 on dist-test-slave-06l8"
I20240502 01:54:48.700752 13279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b3ea4e38a26a477aa838e668484b2338"
format_stamp: "Formatted at 2024-05-02 01:54:48 on dist-test-slave-06l8"
I20240502 01:54:48.707046 13279 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20240502 01:54:48.710500 13293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:48.711831 13279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240502 01:54:48.711990 13279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b3ea4e38a26a477aa838e668484b2338"
format_stamp: "Formatted at 2024-05-02 01:54:48 on dist-test-slave-06l8"
I20240502 01:54:48.712183 13279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:48.725973 13279 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:48.726786 13279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:48.727053 13279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:48.728006 13279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:48.729550 13279 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:48.729643 13279 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:48.729756 13279 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:48.729840 13279 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:48.759507 13279 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.193:43203
I20240502 01:54:48.759567 13406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.193:43203 every 8 connection(s)
I20240502 01:54:48.760818 13279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:54:48.770943 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 13279
I20240502 01:54:48.771188 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:54:48.774220 13407 heartbeater.cc:347] Connected to a master server at 127.31.161.254:34193
I20240502 01:54:48.774544 13407 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:48.774780 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.194:0
--local_ip_for_outbound_sockets=127.31.161.194
--webserver_interface=127.31.161.194
--webserver_port=0
--tserver_master_addrs=127.31.161.254:34193
--builtin_ntp_servers=127.31.161.212:43169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:48.775473 13407 heartbeater.cc:511] Master 127.31.161.254:34193 requested a full tablet report, sending...
I20240502 01:54:48.777865 13217 ts_manager.cc:194] Registered new tserver with Master: b3ea4e38a26a477aa838e668484b2338 (127.31.161.193:43203)
I20240502 01:54:48.779819 13217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.193:57851
W20240502 01:54:49.109668 13410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:49.109977 13410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:49.110117 13410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:49.117543 13410 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:49.117682 13410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:49.117837 13410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.194
I20240502 01:54:49.129912 13410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:43169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.31.161.194
--webserver_port=0
--tserver_master_addrs=127.31.161.254:34193
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.194
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:49.131378 13410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:49.133821 13410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:49.147898 13415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:49.157369 13416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:49.157539 13410 server_base.cc:1034] running on GCE node
W20240502 01:54:49.158609 13418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:49.159338 13410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:49.160343 13410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:49.161579 13410 hybrid_clock.cc:648] HybridClock initialized: now 1714614889161498 us; error 67 us; skew 500 ppm
I20240502 01:54:49.161937 13410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:49.164211 13410 webserver.cc:460] Webserver started at http://127.31.161.194:46017/ using document root <none> and password file <none>
I20240502 01:54:49.165055 13410 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:49.165205 13410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:49.165511 13410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:49.168277 13410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e241a6365a8c4bd79e19fc3562718f90"
format_stamp: "Formatted at 2024-05-02 01:54:49 on dist-test-slave-06l8"
I20240502 01:54:49.169018 13410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e241a6365a8c4bd79e19fc3562718f90"
format_stamp: "Formatted at 2024-05-02 01:54:49 on dist-test-slave-06l8"
I20240502 01:54:49.174187 13410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20240502 01:54:49.177615 13424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:49.178947 13410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:54:49.179188 13410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e241a6365a8c4bd79e19fc3562718f90"
format_stamp: "Formatted at 2024-05-02 01:54:49 on dist-test-slave-06l8"
I20240502 01:54:49.179419 13410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:49.200183 13410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:49.201157 13410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:49.201457 13410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:49.202490 13410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:49.204084 13410 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:49.204174 13410 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:49.204254 13410 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:49.204308 13410 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:49.235250 13410 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.194:42253
I20240502 01:54:49.235282 13537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.194:42253 every 8 connection(s)
I20240502 01:54:49.236995 13410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:54:49.246615 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 13410
I20240502 01:54:49.246810 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:54:49.249545 13538 heartbeater.cc:347] Connected to a master server at 127.31.161.254:34193
I20240502 01:54:49.249837 13538 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:49.250138 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.195:0
--local_ip_for_outbound_sockets=127.31.161.195
--webserver_interface=127.31.161.195
--webserver_port=0
--tserver_master_addrs=127.31.161.254:34193
--builtin_ntp_servers=127.31.161.212:43169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:49.250679 13538 heartbeater.cc:511] Master 127.31.161.254:34193 requested a full tablet report, sending...
I20240502 01:54:49.252067 13219 ts_manager.cc:194] Registered new tserver with Master: e241a6365a8c4bd79e19fc3562718f90 (127.31.161.194:42253)
I20240502 01:54:49.253012 13219 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.194:35719
W20240502 01:54:49.582077 13541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:49.582355 13541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:49.582445 13541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:49.589758 13541 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:49.589905 13541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:49.590058 13541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.195
I20240502 01:54:49.598980 13541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:43169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.31.161.195
--webserver_port=0
--tserver_master_addrs=127.31.161.254:34193
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.195
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:49.600297 13541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:49.602136 13541 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:49.613718 13549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:49.614449 13546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:49.614514 13547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:49.614777 13541 server_base.cc:1034] running on GCE node
I20240502 01:54:49.615351 13541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:49.616516 13541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:49.617722 13541 hybrid_clock.cc:648] HybridClock initialized: now 1714614889617669 us; error 40 us; skew 500 ppm
I20240502 01:54:49.618036 13541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:49.620373 13541 webserver.cc:460] Webserver started at http://127.31.161.195:38587/ using document root <none> and password file <none>
I20240502 01:54:49.621150 13541 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:49.621263 13541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:49.621541 13541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:49.624183 13541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "13ee3155088249cf9b5ea066bea2060f"
format_stamp: "Formatted at 2024-05-02 01:54:49 on dist-test-slave-06l8"
I20240502 01:54:49.624965 13541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "13ee3155088249cf9b5ea066bea2060f"
format_stamp: "Formatted at 2024-05-02 01:54:49 on dist-test-slave-06l8"
I20240502 01:54:49.630115 13541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20240502 01:54:49.633423 13555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:49.634747 13541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:54:49.634878 13541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "13ee3155088249cf9b5ea066bea2060f"
format_stamp: "Formatted at 2024-05-02 01:54:49 on dist-test-slave-06l8"
I20240502 01:54:49.635049 13541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:49.656742 13541 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:49.657685 13541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:49.657928 13541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:49.658859 13541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:49.660318 13541 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:49.660413 13541 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:49.660496 13541 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:49.660550 13541 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:49.690138 13541 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.195:45135
I20240502 01:54:49.690191 13668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.195:45135 every 8 connection(s)
I20240502 01:54:49.691483 13541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:54:49.691777 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 13541
I20240502 01:54:49.691934 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:54:49.695477 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.196:0
--local_ip_for_outbound_sockets=127.31.161.196
--webserver_interface=127.31.161.196
--webserver_port=0
--tserver_master_addrs=127.31.161.254:34193
--builtin_ntp_servers=127.31.161.212:43169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:49.705840 13669 heartbeater.cc:347] Connected to a master server at 127.31.161.254:34193
I20240502 01:54:49.706203 13669 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:49.707207 13669 heartbeater.cc:511] Master 127.31.161.254:34193 requested a full tablet report, sending...
I20240502 01:54:49.708837 13219 ts_manager.cc:194] Registered new tserver with Master: 13ee3155088249cf9b5ea066bea2060f (127.31.161.195:45135)
I20240502 01:54:49.709769 13219 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.195:47309
I20240502 01:54:49.783129 13407 heartbeater.cc:503] Master 127.31.161.254:34193 was elected leader, sending a full tablet report...
W20240502 01:54:50.025282 13672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:50.025589 13672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:50.025723 13672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:50.033277 13672 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:50.033423 13672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:50.033569 13672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.196
I20240502 01:54:50.042623 13672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:43169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.31.161.196
--webserver_port=0
--tserver_master_addrs=127.31.161.254:34193
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.196
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:50.043922 13672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:50.045657 13672 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:50.056995 13680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:50.057621 13678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:50.057708 13677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:50.058197 13672 server_base.cc:1034] running on GCE node
I20240502 01:54:50.058725 13672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:50.059594 13672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:50.060796 13672 hybrid_clock.cc:648] HybridClock initialized: now 1714614890060743 us; error 40 us; skew 500 ppm
I20240502 01:54:50.061125 13672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:50.063290 13672 webserver.cc:460] Webserver started at http://127.31.161.196:44691/ using document root <none> and password file <none>
I20240502 01:54:50.063975 13672 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:50.064090 13672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:50.064386 13672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:50.067005 13672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "93f06932797c4d12987ca2e87321a4c2"
format_stamp: "Formatted at 2024-05-02 01:54:50 on dist-test-slave-06l8"
I20240502 01:54:50.067750 13672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "93f06932797c4d12987ca2e87321a4c2"
format_stamp: "Formatted at 2024-05-02 01:54:50 on dist-test-slave-06l8"
I20240502 01:54:50.072615 13672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240502 01:54:50.075950 13687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:50.077284 13672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:54:50.077420 13672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "93f06932797c4d12987ca2e87321a4c2"
format_stamp: "Formatted at 2024-05-02 01:54:50 on dist-test-slave-06l8"
I20240502 01:54:50.077601 13672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:50.097034 13672 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:50.097875 13672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:50.098119 13672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:50.098994 13672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:50.100385 13672 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:50.100478 13672 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:50.100576 13672 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:50.100646 13672 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:50.130311 13672 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.196:42051
I20240502 01:54:50.130375 13800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.196:42051 every 8 connection(s)
I20240502 01:54:50.131565 13672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:54:50.137456 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 13672
I20240502 01:54:50.137673 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:54:50.144533 13801 heartbeater.cc:347] Connected to a master server at 127.31.161.254:34193
I20240502 01:54:50.144788 13801 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:50.145578 13801 heartbeater.cc:511] Master 127.31.161.254:34193 requested a full tablet report, sending...
I20240502 01:54:50.146829 13219 ts_manager.cc:194] Registered new tserver with Master: 93f06932797c4d12987ca2e87321a4c2 (127.31.161.196:42051)
I20240502 01:54:50.147562 13219 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.196:36053
I20240502 01:54:50.153666 32391 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 01:54:50.169067 13219 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55004:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 01:54:50.195590 13341 tablet_service.cc:1467] Processing CreateTablet for tablet c4ba3df930514cf5813fab31dcca8dfe (DEFAULT_TABLE table=TestTable [id=263fd61d961c4aac901177e2afd49dea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:50.196851 13341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4ba3df930514cf5813fab31dcca8dfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:50.206825 13815 tablet_bootstrap.cc:492] T c4ba3df930514cf5813fab31dcca8dfe P b3ea4e38a26a477aa838e668484b2338: Bootstrap starting.
I20240502 01:54:50.210186 13815 tablet_bootstrap.cc:654] T c4ba3df930514cf5813fab31dcca8dfe P b3ea4e38a26a477aa838e668484b2338: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:50.211256 13815 log.cc:824] T c4ba3df930514cf5813fab31dcca8dfe P b3ea4e38a26a477aa838e668484b2338: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:50.213754 13815 tablet_bootstrap.cc:492] T c4ba3df930514cf5813fab31dcca8dfe P b3ea4e38a26a477aa838e668484b2338: No bootstrap required, opened a new log
I20240502 01:54:50.214011 13815 ts_tablet_manager.cc:1367] T c4ba3df930514cf5813fab31dcca8dfe P b3ea4e38a26a477aa838e668484b2338: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 01:54:50.223012 13815 raft_consensus.cc:348] T c4ba3df930514cf5813fab31dcca8dfe P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:50.223459 13815 raft_consensus.cc:374] T c4ba3df930514cf5813fab31dcca8dfe P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:50.223567 13815 raft_consensus.cc:729] T c4ba3df930514cf5813fab31dcca8dfe P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ea4e38a26a477aa838e668484b2338, State: Initialized, Role: FOLLOWER
I20240502 01:54:50.224303 13815 consensus_queue.cc:260] T c4ba3df930514cf5813fab31dcca8dfe P b3ea4e38a26a477aa838e668484b2338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:50.224581 13815 raft_consensus.cc:388] T c4ba3df930514cf5813fab31dcca8dfe P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:50.224714 13815 raft_consensus.cc:482] T c4ba3df930514cf5813fab31dcca8dfe P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:50.224884 13815 raft_consensus.cc:3037] T c4ba3df930514cf5813fab31dcca8dfe P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:50.227928 13815 raft_consensus.cc:504] T c4ba3df930514cf5813fab31dcca8dfe P b3ea4e38a26a477aa838e668484b2338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:50.228533 13815 leader_election.cc:302] T c4ba3df930514cf5813fab31dcca8dfe P b3ea4e38a26a477aa838e668484b2338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3ea4e38a26a477aa838e668484b2338; no voters:
I20240502 01:54:50.229238 13815 leader_election.cc:288] T c4ba3df930514cf5813fab31dcca8dfe P b3ea4e38a26a477aa838e668484b2338 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:54:50.229372 13817 raft_consensus.cc:2781] T c4ba3df930514cf5813fab31dcca8dfe P b3ea4e38a26a477aa838e668484b2338 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:50.229943 13817 raft_consensus.cc:686] T c4ba3df930514cf5813fab31dcca8dfe P b3ea4e38a26a477aa838e668484b2338 [term 1 LEADER]: Becoming Leader. State: Replica: b3ea4e38a26a477aa838e668484b2338, State: Running, Role: LEADER
I20240502 01:54:50.230508 13817 consensus_queue.cc:237] T c4ba3df930514cf5813fab31dcca8dfe P b3ea4e38a26a477aa838e668484b2338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:50.235159 13815 ts_tablet_manager.cc:1398] T c4ba3df930514cf5813fab31dcca8dfe P b3ea4e38a26a477aa838e668484b2338: Time spent starting tablet: real 0.021s user 0.015s sys 0.004s
I20240502 01:54:50.236965 13219 catalog_manager.cc:5484] T c4ba3df930514cf5813fab31dcca8dfe P b3ea4e38a26a477aa838e668484b2338 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3ea4e38a26a477aa838e668484b2338 (127.31.161.193). New cstate: current_term: 1 leader_uuid: "b3ea4e38a26a477aa838e668484b2338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:50.253408 32391 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 01:54:50.255398 32391 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b3ea4e38a26a477aa838e668484b2338 to finish bootstrapping
I20240502 01:54:50.255877 13538 heartbeater.cc:503] Master 127.31.161.254:34193 was elected leader, sending a full tablet report...
I20240502 01:54:50.263712 32391 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e241a6365a8c4bd79e19fc3562718f90 to finish bootstrapping
I20240502 01:54:50.271927 32391 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 13ee3155088249cf9b5ea066bea2060f to finish bootstrapping
I20240502 01:54:50.279539 32391 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 93f06932797c4d12987ca2e87321a4c2 to finish bootstrapping
I20240502 01:54:50.287657 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 13541
I20240502 01:54:50.288516 13664 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:50.367180 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 13541
I20240502 01:54:50.381667 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 13672
I20240502 01:54:50.382843 13796 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:50.462951 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 13672
I20240502 01:54:54.229300 13219 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55004:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:54.240018 13341 tablet_service.cc:1467] Processing CreateTablet for tablet 11498ca3c6cd4cfcb9864e815522e00a (DEFAULT_TABLE table=rebalance_test_table_0 [id=cdb27578087f4f0d8a8e2c9821c317d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:54.240512 13341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11498ca3c6cd4cfcb9864e815522e00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:54.246505 13835 tablet_bootstrap.cc:492] T 11498ca3c6cd4cfcb9864e815522e00a P b3ea4e38a26a477aa838e668484b2338: Bootstrap starting.
I20240502 01:54:54.248042 13835 tablet_bootstrap.cc:654] T 11498ca3c6cd4cfcb9864e815522e00a P b3ea4e38a26a477aa838e668484b2338: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:54.248987 13471 tablet_service.cc:1467] Processing CreateTablet for tablet 79212abfc81544efa0944c5348329b1f (DEFAULT_TABLE table=rebalance_test_table_0 [id=cdb27578087f4f0d8a8e2c9821c317d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:54.249050 13472 tablet_service.cc:1467] Processing CreateTablet for tablet f78729ea01d1484aabc98c28fa6454d1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=cdb27578087f4f0d8a8e2c9821c317d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:54.250085 13835 tablet_bootstrap.cc:492] T 11498ca3c6cd4cfcb9864e815522e00a P b3ea4e38a26a477aa838e668484b2338: No bootstrap required, opened a new log
I20240502 01:54:54.250237 13835 ts_tablet_manager.cc:1367] T 11498ca3c6cd4cfcb9864e815522e00a P b3ea4e38a26a477aa838e668484b2338: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:54:54.250483 13471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79212abfc81544efa0944c5348329b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:54.250963 13472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f78729ea01d1484aabc98c28fa6454d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:54.251001 13835 raft_consensus.cc:348] T 11498ca3c6cd4cfcb9864e815522e00a P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:54.251134 13835 raft_consensus.cc:374] T 11498ca3c6cd4cfcb9864e815522e00a P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:54.251196 13835 raft_consensus.cc:729] T 11498ca3c6cd4cfcb9864e815522e00a P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ea4e38a26a477aa838e668484b2338, State: Initialized, Role: FOLLOWER
I20240502 01:54:54.251425 13835 consensus_queue.cc:260] T 11498ca3c6cd4cfcb9864e815522e00a P b3ea4e38a26a477aa838e668484b2338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:54.251617 13835 raft_consensus.cc:388] T 11498ca3c6cd4cfcb9864e815522e00a P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:54.251698 13835 raft_consensus.cc:482] T 11498ca3c6cd4cfcb9864e815522e00a P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:54.251804 13835 raft_consensus.cc:3037] T 11498ca3c6cd4cfcb9864e815522e00a P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:54.254158 13835 raft_consensus.cc:504] T 11498ca3c6cd4cfcb9864e815522e00a P b3ea4e38a26a477aa838e668484b2338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:54.254395 13835 leader_election.cc:302] T 11498ca3c6cd4cfcb9864e815522e00a P b3ea4e38a26a477aa838e668484b2338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3ea4e38a26a477aa838e668484b2338; no voters:
I20240502 01:54:54.254858 13835 leader_election.cc:288] T 11498ca3c6cd4cfcb9864e815522e00a P b3ea4e38a26a477aa838e668484b2338 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:54:54.255030 13837 raft_consensus.cc:2781] T 11498ca3c6cd4cfcb9864e815522e00a P b3ea4e38a26a477aa838e668484b2338 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:54.265664 13837 raft_consensus.cc:686] T 11498ca3c6cd4cfcb9864e815522e00a P b3ea4e38a26a477aa838e668484b2338 [term 1 LEADER]: Becoming Leader. State: Replica: b3ea4e38a26a477aa838e668484b2338, State: Running, Role: LEADER
I20240502 01:54:54.266104 13835 ts_tablet_manager.cc:1398] T 11498ca3c6cd4cfcb9864e815522e00a P b3ea4e38a26a477aa838e668484b2338: Time spent starting tablet: real 0.016s user 0.005s sys 0.011s
I20240502 01:54:54.266127 13837 consensus_queue.cc:237] T 11498ca3c6cd4cfcb9864e815522e00a P b3ea4e38a26a477aa838e668484b2338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:54.267077 13840 tablet_bootstrap.cc:492] T f78729ea01d1484aabc98c28fa6454d1 P e241a6365a8c4bd79e19fc3562718f90: Bootstrap starting.
W20240502 01:54:54.269333 13408 tablet.cc:2369] T 11498ca3c6cd4cfcb9864e815522e00a P b3ea4e38a26a477aa838e668484b2338: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:54:54.270217 13840 tablet_bootstrap.cc:654] T f78729ea01d1484aabc98c28fa6454d1 P e241a6365a8c4bd79e19fc3562718f90: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:54.271405 13840 log.cc:824] T f78729ea01d1484aabc98c28fa6454d1 P e241a6365a8c4bd79e19fc3562718f90: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:54.271497 13217 catalog_manager.cc:5484] T 11498ca3c6cd4cfcb9864e815522e00a P b3ea4e38a26a477aa838e668484b2338 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3ea4e38a26a477aa838e668484b2338 (127.31.161.193). New cstate: current_term: 1 leader_uuid: "b3ea4e38a26a477aa838e668484b2338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:54.274103 13840 tablet_bootstrap.cc:492] T f78729ea01d1484aabc98c28fa6454d1 P e241a6365a8c4bd79e19fc3562718f90: No bootstrap required, opened a new log
I20240502 01:54:54.274331 13840 ts_tablet_manager.cc:1367] T f78729ea01d1484aabc98c28fa6454d1 P e241a6365a8c4bd79e19fc3562718f90: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 01:54:54.280349 13840 raft_consensus.cc:348] T f78729ea01d1484aabc98c28fa6454d1 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:54.280628 13840 raft_consensus.cc:374] T f78729ea01d1484aabc98c28fa6454d1 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:54.280705 13840 raft_consensus.cc:729] T f78729ea01d1484aabc98c28fa6454d1 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e241a6365a8c4bd79e19fc3562718f90, State: Initialized, Role: FOLLOWER
I20240502 01:54:54.281235 13840 consensus_queue.cc:260] T f78729ea01d1484aabc98c28fa6454d1 P e241a6365a8c4bd79e19fc3562718f90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:54.281409 13840 raft_consensus.cc:388] T f78729ea01d1484aabc98c28fa6454d1 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:54.281502 13840 raft_consensus.cc:482] T f78729ea01d1484aabc98c28fa6454d1 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:54.281617 13840 raft_consensus.cc:3037] T f78729ea01d1484aabc98c28fa6454d1 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:54.283396 13840 raft_consensus.cc:504] T f78729ea01d1484aabc98c28fa6454d1 P e241a6365a8c4bd79e19fc3562718f90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:54.283818 13840 leader_election.cc:302] T f78729ea01d1484aabc98c28fa6454d1 P e241a6365a8c4bd79e19fc3562718f90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e241a6365a8c4bd79e19fc3562718f90; no voters:
I20240502 01:54:54.284348 13840 leader_election.cc:288] T f78729ea01d1484aabc98c28fa6454d1 P e241a6365a8c4bd79e19fc3562718f90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:54:54.284476 13844 raft_consensus.cc:2781] T f78729ea01d1484aabc98c28fa6454d1 P e241a6365a8c4bd79e19fc3562718f90 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:54.285053 13844 raft_consensus.cc:686] T f78729ea01d1484aabc98c28fa6454d1 P e241a6365a8c4bd79e19fc3562718f90 [term 1 LEADER]: Becoming Leader. State: Replica: e241a6365a8c4bd79e19fc3562718f90, State: Running, Role: LEADER
I20240502 01:54:54.285864 13840 ts_tablet_manager.cc:1398] T f78729ea01d1484aabc98c28fa6454d1 P e241a6365a8c4bd79e19fc3562718f90: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20240502 01:54:54.285867 13844 consensus_queue.cc:237] T f78729ea01d1484aabc98c28fa6454d1 P e241a6365a8c4bd79e19fc3562718f90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:54.286252 13840 tablet_bootstrap.cc:492] T 79212abfc81544efa0944c5348329b1f P e241a6365a8c4bd79e19fc3562718f90: Bootstrap starting.
I20240502 01:54:54.287621 13840 tablet_bootstrap.cc:654] T 79212abfc81544efa0944c5348329b1f P e241a6365a8c4bd79e19fc3562718f90: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:54.290313 13219 catalog_manager.cc:5484] T f78729ea01d1484aabc98c28fa6454d1 P e241a6365a8c4bd79e19fc3562718f90 reported cstate change: term changed from 0 to 1, leader changed from <none> to e241a6365a8c4bd79e19fc3562718f90 (127.31.161.194). New cstate: current_term: 1 leader_uuid: "e241a6365a8c4bd79e19fc3562718f90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:54.291715 13840 tablet_bootstrap.cc:492] T 79212abfc81544efa0944c5348329b1f P e241a6365a8c4bd79e19fc3562718f90: No bootstrap required, opened a new log
I20240502 01:54:54.291882 13840 ts_tablet_manager.cc:1367] T 79212abfc81544efa0944c5348329b1f P e241a6365a8c4bd79e19fc3562718f90: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240502 01:54:54.292802 13840 raft_consensus.cc:348] T 79212abfc81544efa0944c5348329b1f P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:54.292965 13840 raft_consensus.cc:374] T 79212abfc81544efa0944c5348329b1f P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:54.293062 13840 raft_consensus.cc:729] T 79212abfc81544efa0944c5348329b1f P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e241a6365a8c4bd79e19fc3562718f90, State: Initialized, Role: FOLLOWER
I20240502 01:54:54.293334 13840 consensus_queue.cc:260] T 79212abfc81544efa0944c5348329b1f P e241a6365a8c4bd79e19fc3562718f90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:54.293505 13840 raft_consensus.cc:388] T 79212abfc81544efa0944c5348329b1f P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:54.293587 13840 raft_consensus.cc:482] T 79212abfc81544efa0944c5348329b1f P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:54.293711 13840 raft_consensus.cc:3037] T 79212abfc81544efa0944c5348329b1f P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:54.295598 13840 raft_consensus.cc:504] T 79212abfc81544efa0944c5348329b1f P e241a6365a8c4bd79e19fc3562718f90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:54.295835 13840 leader_election.cc:302] T 79212abfc81544efa0944c5348329b1f P e241a6365a8c4bd79e19fc3562718f90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e241a6365a8c4bd79e19fc3562718f90; no voters:
I20240502 01:54:54.296025 13840 leader_election.cc:288] T 79212abfc81544efa0944c5348329b1f P e241a6365a8c4bd79e19fc3562718f90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:54:54.296205 13844 raft_consensus.cc:2781] T 79212abfc81544efa0944c5348329b1f P e241a6365a8c4bd79e19fc3562718f90 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:54.296381 13844 raft_consensus.cc:686] T 79212abfc81544efa0944c5348329b1f P e241a6365a8c4bd79e19fc3562718f90 [term 1 LEADER]: Becoming Leader. State: Replica: e241a6365a8c4bd79e19fc3562718f90, State: Running, Role: LEADER
I20240502 01:54:54.296533 13840 ts_tablet_manager.cc:1398] T 79212abfc81544efa0944c5348329b1f P e241a6365a8c4bd79e19fc3562718f90: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 01:54:54.296605 13844 consensus_queue.cc:237] T 79212abfc81544efa0944c5348329b1f P e241a6365a8c4bd79e19fc3562718f90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:54.298378 13219 catalog_manager.cc:5484] T 79212abfc81544efa0944c5348329b1f P e241a6365a8c4bd79e19fc3562718f90 reported cstate change: term changed from 0 to 1, leader changed from <none> to e241a6365a8c4bd79e19fc3562718f90 (127.31.161.194). New cstate: current_term: 1 leader_uuid: "e241a6365a8c4bd79e19fc3562718f90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:54.791569 13219 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55004:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:54.798796 13341 tablet_service.cc:1467] Processing CreateTablet for tablet b445ccdbc3044a84bd5b6769dda692ec (DEFAULT_TABLE table=rebalance_test_table_1 [id=bd4ffa89b6ff49a6ab55f61701933eea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:54.798861 13472 tablet_service.cc:1467] Processing CreateTablet for tablet 3d092c4ceb2d48ee864189d9f85c9b13 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bd4ffa89b6ff49a6ab55f61701933eea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:54.799255 13472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d092c4ceb2d48ee864189d9f85c9b13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:54.799257 13341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b445ccdbc3044a84bd5b6769dda692ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:54.801694 13340 tablet_service.cc:1467] Processing CreateTablet for tablet a3853eb6aac3423f97c29b9f66a61388 (DEFAULT_TABLE table=rebalance_test_table_1 [id=bd4ffa89b6ff49a6ab55f61701933eea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:54.802026 13340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3853eb6aac3423f97c29b9f66a61388. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:54.805529 13871 tablet_bootstrap.cc:492] T 3d092c4ceb2d48ee864189d9f85c9b13 P e241a6365a8c4bd79e19fc3562718f90: Bootstrap starting.
I20240502 01:54:54.806231 13872 tablet_bootstrap.cc:492] T b445ccdbc3044a84bd5b6769dda692ec P b3ea4e38a26a477aa838e668484b2338: Bootstrap starting.
I20240502 01:54:54.807211 13871 tablet_bootstrap.cc:654] T 3d092c4ceb2d48ee864189d9f85c9b13 P e241a6365a8c4bd79e19fc3562718f90: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:54.807546 13872 tablet_bootstrap.cc:654] T b445ccdbc3044a84bd5b6769dda692ec P b3ea4e38a26a477aa838e668484b2338: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:54.808904 13871 tablet_bootstrap.cc:492] T 3d092c4ceb2d48ee864189d9f85c9b13 P e241a6365a8c4bd79e19fc3562718f90: No bootstrap required, opened a new log
I20240502 01:54:54.809052 13871 ts_tablet_manager.cc:1367] T 3d092c4ceb2d48ee864189d9f85c9b13 P e241a6365a8c4bd79e19fc3562718f90: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:54:54.809259 13872 tablet_bootstrap.cc:492] T b445ccdbc3044a84bd5b6769dda692ec P b3ea4e38a26a477aa838e668484b2338: No bootstrap required, opened a new log
I20240502 01:54:54.809386 13872 ts_tablet_manager.cc:1367] T b445ccdbc3044a84bd5b6769dda692ec P b3ea4e38a26a477aa838e668484b2338: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:54:54.809803 13871 raft_consensus.cc:348] T 3d092c4ceb2d48ee864189d9f85c9b13 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:54.809984 13871 raft_consensus.cc:374] T 3d092c4ceb2d48ee864189d9f85c9b13 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:54.810086 13871 raft_consensus.cc:729] T 3d092c4ceb2d48ee864189d9f85c9b13 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e241a6365a8c4bd79e19fc3562718f90, State: Initialized, Role: FOLLOWER
I20240502 01:54:54.810099 13872 raft_consensus.cc:348] T b445ccdbc3044a84bd5b6769dda692ec P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:54.810249 13872 raft_consensus.cc:374] T b445ccdbc3044a84bd5b6769dda692ec P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:54.810331 13872 raft_consensus.cc:729] T b445ccdbc3044a84bd5b6769dda692ec P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ea4e38a26a477aa838e668484b2338, State: Initialized, Role: FOLLOWER
I20240502 01:54:54.810331 13871 consensus_queue.cc:260] T 3d092c4ceb2d48ee864189d9f85c9b13 P e241a6365a8c4bd79e19fc3562718f90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:54.810509 13871 raft_consensus.cc:388] T 3d092c4ceb2d48ee864189d9f85c9b13 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:54.810590 13871 raft_consensus.cc:482] T 3d092c4ceb2d48ee864189d9f85c9b13 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:54.810561 13872 consensus_queue.cc:260] T b445ccdbc3044a84bd5b6769dda692ec P b3ea4e38a26a477aa838e668484b2338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:54.810698 13871 raft_consensus.cc:3037] T 3d092c4ceb2d48ee864189d9f85c9b13 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:54.810746 13872 raft_consensus.cc:388] T b445ccdbc3044a84bd5b6769dda692ec P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:54.810827 13872 raft_consensus.cc:482] T b445ccdbc3044a84bd5b6769dda692ec P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:54.810933 13872 raft_consensus.cc:3037] T b445ccdbc3044a84bd5b6769dda692ec P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:54.813313 13871 raft_consensus.cc:504] T 3d092c4ceb2d48ee864189d9f85c9b13 P e241a6365a8c4bd79e19fc3562718f90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:54.813393 13872 raft_consensus.cc:504] T b445ccdbc3044a84bd5b6769dda692ec P b3ea4e38a26a477aa838e668484b2338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:54.813566 13871 leader_election.cc:302] T 3d092c4ceb2d48ee864189d9f85c9b13 P e241a6365a8c4bd79e19fc3562718f90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e241a6365a8c4bd79e19fc3562718f90; no voters:
I20240502 01:54:54.813601 13872 leader_election.cc:302] T b445ccdbc3044a84bd5b6769dda692ec P b3ea4e38a26a477aa838e668484b2338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3ea4e38a26a477aa838e668484b2338; no voters:
I20240502 01:54:54.813776 13871 leader_election.cc:288] T 3d092c4ceb2d48ee864189d9f85c9b13 P e241a6365a8c4bd79e19fc3562718f90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:54:54.813872 13844 raft_consensus.cc:2781] T 3d092c4ceb2d48ee864189d9f85c9b13 P e241a6365a8c4bd79e19fc3562718f90 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:54.814020 13872 leader_election.cc:288] T b445ccdbc3044a84bd5b6769dda692ec P b3ea4e38a26a477aa838e668484b2338 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:54:54.814186 13875 raft_consensus.cc:2781] T b445ccdbc3044a84bd5b6769dda692ec P b3ea4e38a26a477aa838e668484b2338 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:54.826395 13844 raft_consensus.cc:686] T 3d092c4ceb2d48ee864189d9f85c9b13 P e241a6365a8c4bd79e19fc3562718f90 [term 1 LEADER]: Becoming Leader. State: Replica: e241a6365a8c4bd79e19fc3562718f90, State: Running, Role: LEADER
I20240502 01:54:54.826790 13844 consensus_queue.cc:237] T 3d092c4ceb2d48ee864189d9f85c9b13 P e241a6365a8c4bd79e19fc3562718f90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:54.830219 13875 raft_consensus.cc:686] T b445ccdbc3044a84bd5b6769dda692ec P b3ea4e38a26a477aa838e668484b2338 [term 1 LEADER]: Becoming Leader. State: Replica: b3ea4e38a26a477aa838e668484b2338, State: Running, Role: LEADER
I20240502 01:54:54.830596 13872 ts_tablet_manager.cc:1398] T b445ccdbc3044a84bd5b6769dda692ec P b3ea4e38a26a477aa838e668484b2338: Time spent starting tablet: real 0.021s user 0.005s sys 0.015s
I20240502 01:54:54.830603 13875 consensus_queue.cc:237] T b445ccdbc3044a84bd5b6769dda692ec P b3ea4e38a26a477aa838e668484b2338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:54.830921 13872 tablet_bootstrap.cc:492] T a3853eb6aac3423f97c29b9f66a61388 P b3ea4e38a26a477aa838e668484b2338: Bootstrap starting.
I20240502 01:54:54.830857 13219 catalog_manager.cc:5484] T 3d092c4ceb2d48ee864189d9f85c9b13 P e241a6365a8c4bd79e19fc3562718f90 reported cstate change: term changed from 0 to 1, leader changed from <none> to e241a6365a8c4bd79e19fc3562718f90 (127.31.161.194). New cstate: current_term: 1 leader_uuid: "e241a6365a8c4bd79e19fc3562718f90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:54.832067 13871 ts_tablet_manager.cc:1398] T 3d092c4ceb2d48ee864189d9f85c9b13 P e241a6365a8c4bd79e19fc3562718f90: Time spent starting tablet: real 0.023s user 0.012s sys 0.008s
I20240502 01:54:54.832546 13872 tablet_bootstrap.cc:654] T a3853eb6aac3423f97c29b9f66a61388 P b3ea4e38a26a477aa838e668484b2338: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:54.833096 13217 catalog_manager.cc:5484] T b445ccdbc3044a84bd5b6769dda692ec P b3ea4e38a26a477aa838e668484b2338 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3ea4e38a26a477aa838e668484b2338 (127.31.161.193). New cstate: current_term: 1 leader_uuid: "b3ea4e38a26a477aa838e668484b2338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:54.837682 13872 tablet_bootstrap.cc:492] T a3853eb6aac3423f97c29b9f66a61388 P b3ea4e38a26a477aa838e668484b2338: No bootstrap required, opened a new log
I20240502 01:54:54.837834 13872 ts_tablet_manager.cc:1367] T a3853eb6aac3423f97c29b9f66a61388 P b3ea4e38a26a477aa838e668484b2338: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20240502 01:54:54.838639 13872 raft_consensus.cc:348] T a3853eb6aac3423f97c29b9f66a61388 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:54.838814 13872 raft_consensus.cc:374] T a3853eb6aac3423f97c29b9f66a61388 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:54.838884 13872 raft_consensus.cc:729] T a3853eb6aac3423f97c29b9f66a61388 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ea4e38a26a477aa838e668484b2338, State: Initialized, Role: FOLLOWER
I20240502 01:54:54.839087 13872 consensus_queue.cc:260] T a3853eb6aac3423f97c29b9f66a61388 P b3ea4e38a26a477aa838e668484b2338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:54.839251 13872 raft_consensus.cc:388] T a3853eb6aac3423f97c29b9f66a61388 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:54.839355 13872 raft_consensus.cc:482] T a3853eb6aac3423f97c29b9f66a61388 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:54.839452 13872 raft_consensus.cc:3037] T a3853eb6aac3423f97c29b9f66a61388 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:54.841187 13872 raft_consensus.cc:504] T a3853eb6aac3423f97c29b9f66a61388 P b3ea4e38a26a477aa838e668484b2338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:54.841377 13872 leader_election.cc:302] T a3853eb6aac3423f97c29b9f66a61388 P b3ea4e38a26a477aa838e668484b2338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3ea4e38a26a477aa838e668484b2338; no voters:
I20240502 01:54:54.841542 13872 leader_election.cc:288] T a3853eb6aac3423f97c29b9f66a61388 P b3ea4e38a26a477aa838e668484b2338 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:54:54.841603 13875 raft_consensus.cc:2781] T a3853eb6aac3423f97c29b9f66a61388 P b3ea4e38a26a477aa838e668484b2338 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:54.841776 13875 raft_consensus.cc:686] T a3853eb6aac3423f97c29b9f66a61388 P b3ea4e38a26a477aa838e668484b2338 [term 1 LEADER]: Becoming Leader. State: Replica: b3ea4e38a26a477aa838e668484b2338, State: Running, Role: LEADER
I20240502 01:54:54.841979 13875 consensus_queue.cc:237] T a3853eb6aac3423f97c29b9f66a61388 P b3ea4e38a26a477aa838e668484b2338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:54.842039 13872 ts_tablet_manager.cc:1398] T a3853eb6aac3423f97c29b9f66a61388 P b3ea4e38a26a477aa838e668484b2338: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 01:54:54.843786 13217 catalog_manager.cc:5484] T a3853eb6aac3423f97c29b9f66a61388 P b3ea4e38a26a477aa838e668484b2338 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3ea4e38a26a477aa838e668484b2338 (127.31.161.193). New cstate: current_term: 1 leader_uuid: "b3ea4e38a26a477aa838e668484b2338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:55.343879 13217 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55004:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:55.352766 13472 tablet_service.cc:1467] Processing CreateTablet for tablet 473110a745584308b301f9f54338f164 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4c1d8122b3614459ae5d9857fb9fd115]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:55.352834 13471 tablet_service.cc:1467] Processing CreateTablet for tablet 6ac4b73957b64554981cfa86a9063979 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4c1d8122b3614459ae5d9857fb9fd115]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:55.352957 13341 tablet_service.cc:1467] Processing CreateTablet for tablet 5585aea619c04922a0b8e48101b84872 (DEFAULT_TABLE table=rebalance_test_table_2 [id=4c1d8122b3614459ae5d9857fb9fd115]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:55.353422 13472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 473110a745584308b301f9f54338f164. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:55.353422 13341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5585aea619c04922a0b8e48101b84872. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:55.357316 13471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ac4b73957b64554981cfa86a9063979. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:55.359865 13910 tablet_bootstrap.cc:492] T 5585aea619c04922a0b8e48101b84872 P b3ea4e38a26a477aa838e668484b2338: Bootstrap starting.
I20240502 01:54:55.361591 13910 tablet_bootstrap.cc:654] T 5585aea619c04922a0b8e48101b84872 P b3ea4e38a26a477aa838e668484b2338: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:55.363735 13910 tablet_bootstrap.cc:492] T 5585aea619c04922a0b8e48101b84872 P b3ea4e38a26a477aa838e668484b2338: No bootstrap required, opened a new log
I20240502 01:54:55.363863 13910 ts_tablet_manager.cc:1367] T 5585aea619c04922a0b8e48101b84872 P b3ea4e38a26a477aa838e668484b2338: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:54:55.364444 13910 raft_consensus.cc:348] T 5585aea619c04922a0b8e48101b84872 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:55.364586 13910 raft_consensus.cc:374] T 5585aea619c04922a0b8e48101b84872 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:55.364675 13910 raft_consensus.cc:729] T 5585aea619c04922a0b8e48101b84872 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ea4e38a26a477aa838e668484b2338, State: Initialized, Role: FOLLOWER
I20240502 01:54:55.364943 13910 consensus_queue.cc:260] T 5585aea619c04922a0b8e48101b84872 P b3ea4e38a26a477aa838e668484b2338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:55.365146 13910 raft_consensus.cc:388] T 5585aea619c04922a0b8e48101b84872 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:55.365232 13910 raft_consensus.cc:482] T 5585aea619c04922a0b8e48101b84872 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:55.365370 13910 raft_consensus.cc:3037] T 5585aea619c04922a0b8e48101b84872 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:55.367501 13910 raft_consensus.cc:504] T 5585aea619c04922a0b8e48101b84872 P b3ea4e38a26a477aa838e668484b2338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:55.367717 13910 leader_election.cc:302] T 5585aea619c04922a0b8e48101b84872 P b3ea4e38a26a477aa838e668484b2338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3ea4e38a26a477aa838e668484b2338; no voters:
I20240502 01:54:55.367897 13909 tablet_bootstrap.cc:492] T 473110a745584308b301f9f54338f164 P e241a6365a8c4bd79e19fc3562718f90: Bootstrap starting.
I20240502 01:54:55.367894 13910 leader_election.cc:288] T 5585aea619c04922a0b8e48101b84872 P b3ea4e38a26a477aa838e668484b2338 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:54:55.367983 13875 raft_consensus.cc:2781] T 5585aea619c04922a0b8e48101b84872 P b3ea4e38a26a477aa838e668484b2338 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:55.368789 13875 raft_consensus.cc:686] T 5585aea619c04922a0b8e48101b84872 P b3ea4e38a26a477aa838e668484b2338 [term 1 LEADER]: Becoming Leader. State: Replica: b3ea4e38a26a477aa838e668484b2338, State: Running, Role: LEADER
I20240502 01:54:55.369015 13910 ts_tablet_manager.cc:1398] T 5585aea619c04922a0b8e48101b84872 P b3ea4e38a26a477aa838e668484b2338: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 01:54:55.369060 13875 consensus_queue.cc:237] T 5585aea619c04922a0b8e48101b84872 P b3ea4e38a26a477aa838e668484b2338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:55.369474 13909 tablet_bootstrap.cc:654] T 473110a745584308b301f9f54338f164 P e241a6365a8c4bd79e19fc3562718f90: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:55.371331 13909 tablet_bootstrap.cc:492] T 473110a745584308b301f9f54338f164 P e241a6365a8c4bd79e19fc3562718f90: No bootstrap required, opened a new log
I20240502 01:54:55.371501 13909 ts_tablet_manager.cc:1367] T 473110a745584308b301f9f54338f164 P e241a6365a8c4bd79e19fc3562718f90: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:54:55.371784 13217 catalog_manager.cc:5484] T 5585aea619c04922a0b8e48101b84872 P b3ea4e38a26a477aa838e668484b2338 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3ea4e38a26a477aa838e668484b2338 (127.31.161.193). New cstate: current_term: 1 leader_uuid: "b3ea4e38a26a477aa838e668484b2338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:55.372272 13909 raft_consensus.cc:348] T 473110a745584308b301f9f54338f164 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:55.372447 13909 raft_consensus.cc:374] T 473110a745584308b301f9f54338f164 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:55.372545 13909 raft_consensus.cc:729] T 473110a745584308b301f9f54338f164 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e241a6365a8c4bd79e19fc3562718f90, State: Initialized, Role: FOLLOWER
I20240502 01:54:55.372803 13909 consensus_queue.cc:260] T 473110a745584308b301f9f54338f164 P e241a6365a8c4bd79e19fc3562718f90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:55.372964 13909 raft_consensus.cc:388] T 473110a745584308b301f9f54338f164 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:55.373037 13909 raft_consensus.cc:482] T 473110a745584308b301f9f54338f164 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:55.373157 13909 raft_consensus.cc:3037] T 473110a745584308b301f9f54338f164 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:55.375166 13909 raft_consensus.cc:504] T 473110a745584308b301f9f54338f164 P e241a6365a8c4bd79e19fc3562718f90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:55.375387 13909 leader_election.cc:302] T 473110a745584308b301f9f54338f164 P e241a6365a8c4bd79e19fc3562718f90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e241a6365a8c4bd79e19fc3562718f90; no voters:
I20240502 01:54:55.375825 13909 leader_election.cc:288] T 473110a745584308b301f9f54338f164 P e241a6365a8c4bd79e19fc3562718f90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:54:55.375979 13916 raft_consensus.cc:2781] T 473110a745584308b301f9f54338f164 P e241a6365a8c4bd79e19fc3562718f90 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:55.376441 13916 raft_consensus.cc:686] T 473110a745584308b301f9f54338f164 P e241a6365a8c4bd79e19fc3562718f90 [term 1 LEADER]: Becoming Leader. State: Replica: e241a6365a8c4bd79e19fc3562718f90, State: Running, Role: LEADER
I20240502 01:54:55.376776 13916 consensus_queue.cc:237] T 473110a745584308b301f9f54338f164 P e241a6365a8c4bd79e19fc3562718f90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:55.378998 13219 catalog_manager.cc:5484] T 473110a745584308b301f9f54338f164 P e241a6365a8c4bd79e19fc3562718f90 reported cstate change: term changed from 0 to 1, leader changed from <none> to e241a6365a8c4bd79e19fc3562718f90 (127.31.161.194). New cstate: current_term: 1 leader_uuid: "e241a6365a8c4bd79e19fc3562718f90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:55.382547 13909 ts_tablet_manager.cc:1398] T 473110a745584308b301f9f54338f164 P e241a6365a8c4bd79e19fc3562718f90: Time spent starting tablet: real 0.011s user 0.004s sys 0.006s
I20240502 01:54:55.382786 13909 tablet_bootstrap.cc:492] T 6ac4b73957b64554981cfa86a9063979 P e241a6365a8c4bd79e19fc3562718f90: Bootstrap starting.
I20240502 01:54:55.384117 13909 tablet_bootstrap.cc:654] T 6ac4b73957b64554981cfa86a9063979 P e241a6365a8c4bd79e19fc3562718f90: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:55.385772 13909 tablet_bootstrap.cc:492] T 6ac4b73957b64554981cfa86a9063979 P e241a6365a8c4bd79e19fc3562718f90: No bootstrap required, opened a new log
I20240502 01:54:55.385926 13909 ts_tablet_manager.cc:1367] T 6ac4b73957b64554981cfa86a9063979 P e241a6365a8c4bd79e19fc3562718f90: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20240502 01:54:55.386540 13909 raft_consensus.cc:348] T 6ac4b73957b64554981cfa86a9063979 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:55.386775 13909 raft_consensus.cc:374] T 6ac4b73957b64554981cfa86a9063979 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:55.386844 13909 raft_consensus.cc:729] T 6ac4b73957b64554981cfa86a9063979 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e241a6365a8c4bd79e19fc3562718f90, State: Initialized, Role: FOLLOWER
I20240502 01:54:55.387025 13909 consensus_queue.cc:260] T 6ac4b73957b64554981cfa86a9063979 P e241a6365a8c4bd79e19fc3562718f90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:55.387203 13909 raft_consensus.cc:388] T 6ac4b73957b64554981cfa86a9063979 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:55.387302 13909 raft_consensus.cc:482] T 6ac4b73957b64554981cfa86a9063979 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:55.387392 13909 raft_consensus.cc:3037] T 6ac4b73957b64554981cfa86a9063979 P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:55.389108 13909 raft_consensus.cc:504] T 6ac4b73957b64554981cfa86a9063979 P e241a6365a8c4bd79e19fc3562718f90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:55.389312 13909 leader_election.cc:302] T 6ac4b73957b64554981cfa86a9063979 P e241a6365a8c4bd79e19fc3562718f90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e241a6365a8c4bd79e19fc3562718f90; no voters:
I20240502 01:54:55.389487 13909 leader_election.cc:288] T 6ac4b73957b64554981cfa86a9063979 P e241a6365a8c4bd79e19fc3562718f90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:54:55.389575 13917 raft_consensus.cc:2781] T 6ac4b73957b64554981cfa86a9063979 P e241a6365a8c4bd79e19fc3562718f90 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:55.389750 13917 raft_consensus.cc:686] T 6ac4b73957b64554981cfa86a9063979 P e241a6365a8c4bd79e19fc3562718f90 [term 1 LEADER]: Becoming Leader. State: Replica: e241a6365a8c4bd79e19fc3562718f90, State: Running, Role: LEADER
I20240502 01:54:55.389911 13909 ts_tablet_manager.cc:1398] T 6ac4b73957b64554981cfa86a9063979 P e241a6365a8c4bd79e19fc3562718f90: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20240502 01:54:55.390066 13917 consensus_queue.cc:237] T 6ac4b73957b64554981cfa86a9063979 P e241a6365a8c4bd79e19fc3562718f90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:55.392160 13219 catalog_manager.cc:5484] T 6ac4b73957b64554981cfa86a9063979 P e241a6365a8c4bd79e19fc3562718f90 reported cstate change: term changed from 0 to 1, leader changed from <none> to e241a6365a8c4bd79e19fc3562718f90 (127.31.161.194). New cstate: current_term: 1 leader_uuid: "e241a6365a8c4bd79e19fc3562718f90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:55.883242 13219 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55004:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 01:54:55.891018 13472 tablet_service.cc:1467] Processing CreateTablet for tablet 1a6431def0c84cb398bd90310459a5df (DEFAULT_TABLE table=rebalance_test_table_3 [id=a27303d780c44d2db1c3cd1c58cdcd76]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:55.891028 13341 tablet_service.cc:1467] Processing CreateTablet for tablet 7d8f8dd3d195478b9a41cb7ebd1c92a9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a27303d780c44d2db1c3cd1c58cdcd76]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:55.891448 13472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a6431def0c84cb398bd90310459a5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:55.891455 13341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d8f8dd3d195478b9a41cb7ebd1c92a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:55.891980 13340 tablet_service.cc:1467] Processing CreateTablet for tablet 9311c6ddb7fe46c2affc24c9289e86f1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a27303d780c44d2db1c3cd1c58cdcd76]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:54:55.892338 13340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9311c6ddb7fe46c2affc24c9289e86f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:55.897397 13945 tablet_bootstrap.cc:492] T 1a6431def0c84cb398bd90310459a5df P e241a6365a8c4bd79e19fc3562718f90: Bootstrap starting.
I20240502 01:54:55.898730 13946 tablet_bootstrap.cc:492] T 7d8f8dd3d195478b9a41cb7ebd1c92a9 P b3ea4e38a26a477aa838e668484b2338: Bootstrap starting.
I20240502 01:54:55.898792 13945 tablet_bootstrap.cc:654] T 1a6431def0c84cb398bd90310459a5df P e241a6365a8c4bd79e19fc3562718f90: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:55.900014 13946 tablet_bootstrap.cc:654] T 7d8f8dd3d195478b9a41cb7ebd1c92a9 P b3ea4e38a26a477aa838e668484b2338: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:55.900545 13945 tablet_bootstrap.cc:492] T 1a6431def0c84cb398bd90310459a5df P e241a6365a8c4bd79e19fc3562718f90: No bootstrap required, opened a new log
I20240502 01:54:55.900683 13945 ts_tablet_manager.cc:1367] T 1a6431def0c84cb398bd90310459a5df P e241a6365a8c4bd79e19fc3562718f90: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:54:55.901336 13945 raft_consensus.cc:348] T 1a6431def0c84cb398bd90310459a5df P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:55.901485 13945 raft_consensus.cc:374] T 1a6431def0c84cb398bd90310459a5df P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:55.901566 13945 raft_consensus.cc:729] T 1a6431def0c84cb398bd90310459a5df P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e241a6365a8c4bd79e19fc3562718f90, State: Initialized, Role: FOLLOWER
I20240502 01:54:55.901754 13945 consensus_queue.cc:260] T 1a6431def0c84cb398bd90310459a5df P e241a6365a8c4bd79e19fc3562718f90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:55.901883 13946 tablet_bootstrap.cc:492] T 7d8f8dd3d195478b9a41cb7ebd1c92a9 P b3ea4e38a26a477aa838e668484b2338: No bootstrap required, opened a new log
I20240502 01:54:55.901904 13945 raft_consensus.cc:388] T 1a6431def0c84cb398bd90310459a5df P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:55.901979 13945 raft_consensus.cc:482] T 1a6431def0c84cb398bd90310459a5df P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:55.902016 13946 ts_tablet_manager.cc:1367] T 7d8f8dd3d195478b9a41cb7ebd1c92a9 P b3ea4e38a26a477aa838e668484b2338: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:54:55.902081 13945 raft_consensus.cc:3037] T 1a6431def0c84cb398bd90310459a5df P e241a6365a8c4bd79e19fc3562718f90 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:55.902585 13946 raft_consensus.cc:348] T 7d8f8dd3d195478b9a41cb7ebd1c92a9 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:55.902729 13946 raft_consensus.cc:374] T 7d8f8dd3d195478b9a41cb7ebd1c92a9 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:55.902817 13946 raft_consensus.cc:729] T 7d8f8dd3d195478b9a41cb7ebd1c92a9 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ea4e38a26a477aa838e668484b2338, State: Initialized, Role: FOLLOWER
I20240502 01:54:55.903031 13946 consensus_queue.cc:260] T 7d8f8dd3d195478b9a41cb7ebd1c92a9 P b3ea4e38a26a477aa838e668484b2338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:55.903187 13946 raft_consensus.cc:388] T 7d8f8dd3d195478b9a41cb7ebd1c92a9 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:55.903261 13946 raft_consensus.cc:482] T 7d8f8dd3d195478b9a41cb7ebd1c92a9 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:55.903360 13946 raft_consensus.cc:3037] T 7d8f8dd3d195478b9a41cb7ebd1c92a9 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:55.904080 13945 raft_consensus.cc:504] T 1a6431def0c84cb398bd90310459a5df P e241a6365a8c4bd79e19fc3562718f90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:55.904347 13945 leader_election.cc:302] T 1a6431def0c84cb398bd90310459a5df P e241a6365a8c4bd79e19fc3562718f90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e241a6365a8c4bd79e19fc3562718f90; no voters:
I20240502 01:54:55.904568 13945 leader_election.cc:288] T 1a6431def0c84cb398bd90310459a5df P e241a6365a8c4bd79e19fc3562718f90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:54:55.904665 13917 raft_consensus.cc:2781] T 1a6431def0c84cb398bd90310459a5df P e241a6365a8c4bd79e19fc3562718f90 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:55.905216 13917 raft_consensus.cc:686] T 1a6431def0c84cb398bd90310459a5df P e241a6365a8c4bd79e19fc3562718f90 [term 1 LEADER]: Becoming Leader. State: Replica: e241a6365a8c4bd79e19fc3562718f90, State: Running, Role: LEADER
I20240502 01:54:55.905463 13917 consensus_queue.cc:237] T 1a6431def0c84cb398bd90310459a5df P e241a6365a8c4bd79e19fc3562718f90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } }
I20240502 01:54:55.905679 13946 raft_consensus.cc:504] T 7d8f8dd3d195478b9a41cb7ebd1c92a9 P b3ea4e38a26a477aa838e668484b2338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:55.905911 13946 leader_election.cc:302] T 7d8f8dd3d195478b9a41cb7ebd1c92a9 P b3ea4e38a26a477aa838e668484b2338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3ea4e38a26a477aa838e668484b2338; no voters:
I20240502 01:54:55.908166 13219 catalog_manager.cc:5484] T 1a6431def0c84cb398bd90310459a5df P e241a6365a8c4bd79e19fc3562718f90 reported cstate change: term changed from 0 to 1, leader changed from <none> to e241a6365a8c4bd79e19fc3562718f90 (127.31.161.194). New cstate: current_term: 1 leader_uuid: "e241a6365a8c4bd79e19fc3562718f90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e241a6365a8c4bd79e19fc3562718f90" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 42253 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:55.909322 13946 leader_election.cc:288] T 7d8f8dd3d195478b9a41cb7ebd1c92a9 P b3ea4e38a26a477aa838e668484b2338 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:54:55.909471 13952 raft_consensus.cc:2781] T 7d8f8dd3d195478b9a41cb7ebd1c92a9 P b3ea4e38a26a477aa838e668484b2338 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:55.910084 13946 ts_tablet_manager.cc:1398] T 7d8f8dd3d195478b9a41cb7ebd1c92a9 P b3ea4e38a26a477aa838e668484b2338: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20240502 01:54:55.910327 13946 tablet_bootstrap.cc:492] T 9311c6ddb7fe46c2affc24c9289e86f1 P b3ea4e38a26a477aa838e668484b2338: Bootstrap starting.
I20240502 01:54:55.911553 13946 tablet_bootstrap.cc:654] T 9311c6ddb7fe46c2affc24c9289e86f1 P b3ea4e38a26a477aa838e668484b2338: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:55.912660 13945 ts_tablet_manager.cc:1398] T 1a6431def0c84cb398bd90310459a5df P e241a6365a8c4bd79e19fc3562718f90: Time spent starting tablet: real 0.012s user 0.004s sys 0.007s
I20240502 01:54:55.912886 13952 raft_consensus.cc:686] T 7d8f8dd3d195478b9a41cb7ebd1c92a9 P b3ea4e38a26a477aa838e668484b2338 [term 1 LEADER]: Becoming Leader. State: Replica: b3ea4e38a26a477aa838e668484b2338, State: Running, Role: LEADER
I20240502 01:54:55.913156 13952 consensus_queue.cc:237] T 7d8f8dd3d195478b9a41cb7ebd1c92a9 P b3ea4e38a26a477aa838e668484b2338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:55.913810 13946 tablet_bootstrap.cc:492] T 9311c6ddb7fe46c2affc24c9289e86f1 P b3ea4e38a26a477aa838e668484b2338: No bootstrap required, opened a new log
I20240502 01:54:55.914028 13946 ts_tablet_manager.cc:1367] T 9311c6ddb7fe46c2affc24c9289e86f1 P b3ea4e38a26a477aa838e668484b2338: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:54:55.914788 13946 raft_consensus.cc:348] T 9311c6ddb7fe46c2affc24c9289e86f1 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:55.915059 13946 raft_consensus.cc:374] T 9311c6ddb7fe46c2affc24c9289e86f1 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:55.915158 13946 raft_consensus.cc:729] T 9311c6ddb7fe46c2affc24c9289e86f1 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ea4e38a26a477aa838e668484b2338, State: Initialized, Role: FOLLOWER
I20240502 01:54:55.915374 13946 consensus_queue.cc:260] T 9311c6ddb7fe46c2affc24c9289e86f1 P b3ea4e38a26a477aa838e668484b2338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:55.915519 13946 raft_consensus.cc:388] T 9311c6ddb7fe46c2affc24c9289e86f1 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:55.915616 13946 raft_consensus.cc:482] T 9311c6ddb7fe46c2affc24c9289e86f1 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:55.915714 13946 raft_consensus.cc:3037] T 9311c6ddb7fe46c2affc24c9289e86f1 P b3ea4e38a26a477aa838e668484b2338 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:55.916965 13217 catalog_manager.cc:5484] T 7d8f8dd3d195478b9a41cb7ebd1c92a9 P b3ea4e38a26a477aa838e668484b2338 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3ea4e38a26a477aa838e668484b2338 (127.31.161.193). New cstate: current_term: 1 leader_uuid: "b3ea4e38a26a477aa838e668484b2338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:55.918105 13946 raft_consensus.cc:504] T 9311c6ddb7fe46c2affc24c9289e86f1 P b3ea4e38a26a477aa838e668484b2338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:55.918296 13946 leader_election.cc:302] T 9311c6ddb7fe46c2affc24c9289e86f1 P b3ea4e38a26a477aa838e668484b2338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3ea4e38a26a477aa838e668484b2338; no voters:
I20240502 01:54:55.918476 13946 leader_election.cc:288] T 9311c6ddb7fe46c2affc24c9289e86f1 P b3ea4e38a26a477aa838e668484b2338 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:54:55.918519 13952 raft_consensus.cc:2781] T 9311c6ddb7fe46c2affc24c9289e86f1 P b3ea4e38a26a477aa838e668484b2338 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:55.918756 13952 raft_consensus.cc:686] T 9311c6ddb7fe46c2affc24c9289e86f1 P b3ea4e38a26a477aa838e668484b2338 [term 1 LEADER]: Becoming Leader. State: Replica: b3ea4e38a26a477aa838e668484b2338, State: Running, Role: LEADER
I20240502 01:54:55.918921 13946 ts_tablet_manager.cc:1398] T 9311c6ddb7fe46c2affc24c9289e86f1 P b3ea4e38a26a477aa838e668484b2338: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:54:55.919008 13952 consensus_queue.cc:237] T 9311c6ddb7fe46c2affc24c9289e86f1 P b3ea4e38a26a477aa838e668484b2338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } }
I20240502 01:54:55.921128 13217 catalog_manager.cc:5484] T 9311c6ddb7fe46c2affc24c9289e86f1 P b3ea4e38a26a477aa838e668484b2338 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3ea4e38a26a477aa838e668484b2338 (127.31.161.193). New cstate: current_term: 1 leader_uuid: "b3ea4e38a26a477aa838e668484b2338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ea4e38a26a477aa838e668484b2338" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 43203 } health_report { overall_health: HEALTHY } } }
I20240502 01:54:56.405052 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.195:45135
--local_ip_for_outbound_sockets=127.31.161.195
--tserver_master_addrs=127.31.161.254:34193
--webserver_port=38587
--webserver_interface=127.31.161.195
--builtin_ntp_servers=127.31.161.212:43169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:54:56.732379 13980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:56.732694 13980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:56.732834 13980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:56.740285 13980 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:56.740430 13980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:56.740581 13980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.195
I20240502 01:54:56.749418 13980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:43169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.195:45135
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.31.161.195
--webserver_port=38587
--tserver_master_addrs=127.31.161.254:34193
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.195
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:56.750725 13980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:56.752494 13980 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:56.764410 13986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:56.764406 13985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:56.765448 13988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:56.765733 13980 server_base.cc:1034] running on GCE node
I20240502 01:54:56.766453 13980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:56.767567 13980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:56.768792 13980 hybrid_clock.cc:648] HybridClock initialized: now 1714614896768747 us; error 35 us; skew 500 ppm
I20240502 01:54:56.769177 13980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:56.771813 13980 webserver.cc:460] Webserver started at http://127.31.161.195:38587/ using document root <none> and password file <none>
I20240502 01:54:56.772749 13980 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:56.772889 13980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:56.779446 13980 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20240502 01:54:56.782378 13994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:56.783738 13980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:54:56.783895 13980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "13ee3155088249cf9b5ea066bea2060f"
format_stamp: "Formatted at 2024-05-02 01:54:49 on dist-test-slave-06l8"
I20240502 01:54:56.784760 13980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:56.814985 13980 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:56.815816 13980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:56.816093 13980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:56.817019 13980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:56.818588 13980 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:56.818665 13980 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:56.818781 13980 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:56.818832 13980 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:56.847561 13980 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.195:45135
I20240502 01:54:56.847616 14106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.195:45135 every 8 connection(s)
I20240502 01:54:56.848966 13980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:54:56.853562 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 13980
I20240502 01:54:56.854189 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.196:42051
--local_ip_for_outbound_sockets=127.31.161.196
--tserver_master_addrs=127.31.161.254:34193
--webserver_port=44691
--webserver_interface=127.31.161.196
--builtin_ntp_servers=127.31.161.212:43169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:54:56.869905 14107 heartbeater.cc:347] Connected to a master server at 127.31.161.254:34193
I20240502 01:54:56.870252 14107 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:56.871275 14107 heartbeater.cc:511] Master 127.31.161.254:34193 requested a full tablet report, sending...
I20240502 01:54:56.872840 13217 ts_manager.cc:194] Re-registered known tserver with Master: 13ee3155088249cf9b5ea066bea2060f (127.31.161.195:45135)
I20240502 01:54:56.873680 13217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.195:48075
W20240502 01:54:57.189749 14112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:57.190017 14112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:57.190151 14112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:57.197407 14112 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:57.197516 14112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:57.197674 14112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.196
I20240502 01:54:57.206316 14112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:43169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.196:42051
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.31.161.196
--webserver_port=44691
--tserver_master_addrs=127.31.161.254:34193
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.196
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:57.207610 14112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:57.209342 14112 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:57.221585 14118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:57.221588 14120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:57.222434 14117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:57.223652 14112 server_base.cc:1034] running on GCE node
I20240502 01:54:57.224308 14112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:57.225514 14112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:57.226778 14112 hybrid_clock.cc:648] HybridClock initialized: now 1714614897226731 us; error 35 us; skew 500 ppm
I20240502 01:54:57.227145 14112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:57.229349 14112 webserver.cc:460] Webserver started at http://127.31.161.196:44691/ using document root <none> and password file <none>
I20240502 01:54:57.230019 14112 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:57.230131 14112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:57.235139 14112 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20240502 01:54:57.237920 14126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:57.239252 14112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:54:57.239404 14112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "93f06932797c4d12987ca2e87321a4c2"
format_stamp: "Formatted at 2024-05-02 01:54:50 on dist-test-slave-06l8"
I20240502 01:54:57.240170 14112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:57.269659 14112 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:57.270494 14112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:57.270771 14112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:57.271729 14112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:54:57.273238 14112 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:54:57.273316 14112 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:57.273447 14112 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:54:57.273504 14112 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:57.302897 14112 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.196:42051
I20240502 01:54:57.302963 14238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.196:42051 every 8 connection(s)
I20240502 01:54:57.304329 14112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:54:57.307765 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 14112
I20240502 01:54:57.317394 14239 heartbeater.cc:347] Connected to a master server at 127.31.161.254:34193
I20240502 01:54:57.317727 14239 heartbeater.cc:464] Registering TS with master...
I20240502 01:54:57.318765 14239 heartbeater.cc:511] Master 127.31.161.254:34193 requested a full tablet report, sending...
I20240502 01:54:57.320348 13217 ts_manager.cc:194] Re-registered known tserver with Master: 93f06932797c4d12987ca2e87321a4c2 (127.31.161.196:42051)
I20240502 01:54:57.321189 13217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.196:47869
I20240502 01:54:57.828609 14042 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:57.839864 14174 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:57.858598 13341 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:57.876657 14107 heartbeater.cc:503] Master 127.31.161.254:34193 was elected leader, sending a full tablet report...
I20240502 01:54:57.879662 13472 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:58.045624 14042 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:58.050180 14174 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:58.051199 13341 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:58.060681 13472 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:54:58.313503 14042 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:58.323540 14239 heartbeater.cc:503] Master 127.31.161.254:34193 was elected leader, sending a full tablet report...
I20240502 01:54:58.326545 14174 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:54:58.332273 13341 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 01:54:58.344055 13472 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
2e9fbea5b73148cbac695089ef8dc217 | 127.31.161.254:34193 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.31.161.212:43169 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
13ee3155088249cf9b5ea066bea2060f | 127.31.161.195:45135 | HEALTHY | <none> | 0 | 0
93f06932797c4d12987ca2e87321a4c2 | 127.31.161.196:42051 | HEALTHY | <none> | 0 | 0
b3ea4e38a26a477aa838e668484b2338 | 127.31.161.193:43203 | HEALTHY | <none> | 7 | 0
e241a6365a8c4bd79e19fc3562718f90 | 127.31.161.194:42253 | HEALTHY | <none> | 6 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.31.161.193 | experimental | 127.31.161.193:43203
local_ip_for_outbound_sockets | 127.31.161.194 | experimental | 127.31.161.194:42253
local_ip_for_outbound_sockets | 127.31.161.195 | experimental | 127.31.161.195:45135
local_ip_for_outbound_sockets | 127.31.161.196 | experimental | 127.31.161.196:42051
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.31.161.193:43203
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.31.161.194:42253
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.31.161.195:45135
server_dump_info_path | /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.31.161.196:42051
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.31.161.212:43169 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 6
Third Quartile | 7
Maximum | 7
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 5
Tablets | 13
Replicas | 13
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 01:54:58.382647 32391 log_verifier.cc:126] Checking tablet 11498ca3c6cd4cfcb9864e815522e00a
I20240502 01:54:58.390223 32391 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 11498ca3c6cd4cfcb9864e815522e00a
I20240502 01:54:58.390324 32391 log_verifier.cc:126] Checking tablet 1a6431def0c84cb398bd90310459a5df
I20240502 01:54:58.397469 32391 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 1a6431def0c84cb398bd90310459a5df
I20240502 01:54:58.397554 32391 log_verifier.cc:126] Checking tablet 3d092c4ceb2d48ee864189d9f85c9b13
I20240502 01:54:58.404737 32391 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 3d092c4ceb2d48ee864189d9f85c9b13
I20240502 01:54:58.404825 32391 log_verifier.cc:126] Checking tablet 473110a745584308b301f9f54338f164
I20240502 01:54:58.412446 32391 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 473110a745584308b301f9f54338f164
I20240502 01:54:58.412545 32391 log_verifier.cc:126] Checking tablet 5585aea619c04922a0b8e48101b84872
I20240502 01:54:58.420104 32391 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 5585aea619c04922a0b8e48101b84872
I20240502 01:54:58.420197 32391 log_verifier.cc:126] Checking tablet 6ac4b73957b64554981cfa86a9063979
I20240502 01:54:58.427500 32391 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 6ac4b73957b64554981cfa86a9063979
I20240502 01:54:58.427594 32391 log_verifier.cc:126] Checking tablet 79212abfc81544efa0944c5348329b1f
I20240502 01:54:58.434813 32391 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 79212abfc81544efa0944c5348329b1f
I20240502 01:54:58.434907 32391 log_verifier.cc:126] Checking tablet 7d8f8dd3d195478b9a41cb7ebd1c92a9
I20240502 01:54:58.442445 32391 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 7d8f8dd3d195478b9a41cb7ebd1c92a9
I20240502 01:54:58.442543 32391 log_verifier.cc:126] Checking tablet 9311c6ddb7fe46c2affc24c9289e86f1
I20240502 01:54:58.449992 32391 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 9311c6ddb7fe46c2affc24c9289e86f1
I20240502 01:54:58.450096 32391 log_verifier.cc:126] Checking tablet a3853eb6aac3423f97c29b9f66a61388
I20240502 01:54:58.457929 32391 log_verifier.cc:177] Verified matching terms for 1 ops in tablet a3853eb6aac3423f97c29b9f66a61388
I20240502 01:54:58.458025 32391 log_verifier.cc:126] Checking tablet b445ccdbc3044a84bd5b6769dda692ec
I20240502 01:54:58.465601 32391 log_verifier.cc:177] Verified matching terms for 2 ops in tablet b445ccdbc3044a84bd5b6769dda692ec
I20240502 01:54:58.465723 32391 log_verifier.cc:126] Checking tablet c4ba3df930514cf5813fab31dcca8dfe
I20240502 01:54:58.473119 32391 log_verifier.cc:177] Verified matching terms for 1 ops in tablet c4ba3df930514cf5813fab31dcca8dfe
I20240502 01:54:58.473213 32391 log_verifier.cc:126] Checking tablet f78729ea01d1484aabc98c28fa6454d1
I20240502 01:54:58.480741 32391 log_verifier.cc:177] Verified matching terms for 2 ops in tablet f78729ea01d1484aabc98c28fa6454d1
I20240502 01:54:58.481684 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 13279
I20240502 01:54:58.483036 13402 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:58.582119 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 13279
I20240502 01:54:58.601230 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 13410
I20240502 01:54:58.602274 13532 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:58.699450 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 13410
I20240502 01:54:58.717706 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 13980
I20240502 01:54:58.723997 14102 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:58.809791 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 13980
I20240502 01:54:58.825740 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 14112
I20240502 01:54:58.831784 14234 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:58.916693 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 14112
I20240502 01:54:58.931917 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 13187
I20240502 01:54:58.933151 13249 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:54:59.051505 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 13187
2024-05-02T01:54:59Z chronyd exiting
[ OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/5 (11377 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (11377 ms total)
[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN ] LocationAwareRebalancingParamTest.Rebalance/idx7_rf2_t3_l_1_2_1
I20240502 01:54:59.102448 32391 test_util.cc:268] Using random seed: -469970216
I20240502 01:54:59.104548 32391 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:54:59.104611 32391 ts_itest-base.cc:113] --------------
I20240502 01:54:59.104650 32391 ts_itest-base.cc:114] 4 tablet servers
I20240502 01:54:59.104683 32391 ts_itest-base.cc:115] 2 replicas per TS
I20240502 01:54:59.104725 32391 ts_itest-base.cc:116] --------------
2024-05-02T01:54:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:54:59Z Disabled control of system clock
I20240502 01:54:59.154291 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.161.254:36883
--webserver_interface=127.31.161.254
--webserver_port=0
--builtin_ntp_servers=127.31.161.212:39323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.161.254:36883
--location_mapping_cmd=/tmp/dist-test-taskMchxvY/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/location-assignment.state --map /A:1 --map /B:2 --map /C:1
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240502 01:54:59.497829 14353 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 01:54:59.498198 14353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:54:59.498297 14353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:54:59.498400 14353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:54:59.506142 14353 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:54:59.506273 14353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:54:59.506371 14353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:54:59.506433 14353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:54:59.506513 14353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:54:59.515976 14353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:39323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskMchxvY/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/location-assignment.state --map /A:1 --map /B:2 --map /C:1
--ipki_ca_key_size=768
--master_addresses=127.31.161.254:36883
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.161.254:36883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.31.161.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:54:59.517383 14353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:54:59.519258 14353 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:54:59.529978 14358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:59.530637 14359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:54:59.530815 14361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:54:59.531100 14353 server_base.cc:1034] running on GCE node
I20240502 01:54:59.531860 14353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:54:59.533349 14353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:54:59.534601 14353 hybrid_clock.cc:648] HybridClock initialized: now 1714614899534516 us; error 54 us; skew 500 ppm
I20240502 01:54:59.534953 14353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:54:59.537551 14353 webserver.cc:460] Webserver started at http://127.31.161.254:43281/ using document root <none> and password file <none>
I20240502 01:54:59.538309 14353 fs_manager.cc:359] Metadata directory not provided
I20240502 01:54:59.538439 14353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:54:59.538743 14353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:54:59.541496 14353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "c81dd540fb404b30af9fdfffd23e760d"
format_stamp: "Formatted at 2024-05-02 01:54:59 on dist-test-slave-06l8"
I20240502 01:54:59.542275 14353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "c81dd540fb404b30af9fdfffd23e760d"
format_stamp: "Formatted at 2024-05-02 01:54:59 on dist-test-slave-06l8"
I20240502 01:54:59.547495 14353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20240502 01:54:59.551062 14367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:54:59.552345 14353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20240502 01:54:59.552515 14353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
uuid: "c81dd540fb404b30af9fdfffd23e760d"
format_stamp: "Formatted at 2024-05-02 01:54:59 on dist-test-slave-06l8"
I20240502 01:54:59.552699 14353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:54:59.591215 14353 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:54:59.592180 14353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:54:59.592478 14353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:54:59.610656 14353 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.254:36883
I20240502 01:54:59.610687 14419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.254:36883 every 8 connection(s)
I20240502 01:54:59.612071 14353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:54:59.616039 14420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:54:59.617780 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 14353
I20240502 01:54:59.626041 14420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d: Bootstrap starting.
I20240502 01:54:59.629446 14420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d: Neither blocks nor log segments found. Creating new log.
I20240502 01:54:59.630479 14420 log.cc:824] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d: Log is configured to *not* fsync() on all Append() calls
I20240502 01:54:59.633311 14420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d: No bootstrap required, opened a new log
I20240502 01:54:59.640323 14420 raft_consensus.cc:348] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c81dd540fb404b30af9fdfffd23e760d" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 36883 } }
I20240502 01:54:59.640705 14420 raft_consensus.cc:374] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:54:59.640810 14420 raft_consensus.cc:729] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c81dd540fb404b30af9fdfffd23e760d, State: Initialized, Role: FOLLOWER
I20240502 01:54:59.641533 14420 consensus_queue.cc:260] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c81dd540fb404b30af9fdfffd23e760d" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 36883 } }
I20240502 01:54:59.641743 14420 raft_consensus.cc:388] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:54:59.641876 14420 raft_consensus.cc:482] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:54:59.642020 14420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:54:59.644109 14420 raft_consensus.cc:504] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c81dd540fb404b30af9fdfffd23e760d" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 36883 } }
I20240502 01:54:59.644623 14420 leader_election.cc:302] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c81dd540fb404b30af9fdfffd23e760d; no voters:
I20240502 01:54:59.645336 14420 leader_election.cc:288] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:54:59.645447 14423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:54:59.646595 14423 raft_consensus.cc:686] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d [term 1 LEADER]: Becoming Leader. State: Replica: c81dd540fb404b30af9fdfffd23e760d, State: Running, Role: LEADER
I20240502 01:54:59.647267 14423 consensus_queue.cc:237] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c81dd540fb404b30af9fdfffd23e760d" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 36883 } }
I20240502 01:54:59.647707 14420 sys_catalog.cc:564] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:54:59.649969 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:54:59.650197 14425 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c81dd540fb404b30af9fdfffd23e760d. Latest consensus state: current_term: 1 leader_uuid: "c81dd540fb404b30af9fdfffd23e760d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c81dd540fb404b30af9fdfffd23e760d" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 36883 } } }
I20240502 01:54:59.650585 14425 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d [sys.catalog]: This master's current role is: LEADER
I20240502 01:54:59.651155 14424 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c81dd540fb404b30af9fdfffd23e760d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c81dd540fb404b30af9fdfffd23e760d" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 36883 } } }
I20240502 01:54:59.651458 14424 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d [sys.catalog]: This master's current role is: LEADER
I20240502 01:54:59.677585 14428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:54:59.682868 14428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:54:59.694586 14428 catalog_manager.cc:1347] Generated new cluster ID: 43499ece7a24481595a38f5c68561458
I20240502 01:54:59.694797 14428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:54:59.704386 14428 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:54:59.705634 14428 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:54:59.716687 14428 catalog_manager.cc:5857] T 00000000000000000000000000000000 P c81dd540fb404b30af9fdfffd23e760d: Generated new TSK 0
I20240502 01:54:59.717643 14428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:54:59.729460 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.193:0
--local_ip_for_outbound_sockets=127.31.161.193
--webserver_interface=127.31.161.193
--webserver_port=0
--tserver_master_addrs=127.31.161.254:36883
--builtin_ntp_servers=127.31.161.212:39323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:55:00.091091 14446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:00.091369 14446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:00.091467 14446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:00.099125 14446 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:00.099259 14446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:00.099382 14446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.193
I20240502 01:55:00.108637 14446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:39323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.31.161.193
--webserver_port=0
--tserver_master_addrs=127.31.161.254:36883
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.193
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:00.109997 14446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:00.111665 14446 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:00.122768 14451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:00.123318 14452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:00.123754 14454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:00.123785 14446 server_base.cc:1034] running on GCE node
I20240502 01:55:00.124394 14446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:00.125313 14446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:00.126502 14446 hybrid_clock.cc:648] HybridClock initialized: now 1714614900126449 us; error 44 us; skew 500 ppm
I20240502 01:55:00.126785 14446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:00.128857 14446 webserver.cc:460] Webserver started at http://127.31.161.193:38377/ using document root <none> and password file <none>
I20240502 01:55:00.129578 14446 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:00.129671 14446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:00.129940 14446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:00.132406 14446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "382625cd16224c8aa404242500ab3bfa"
format_stamp: "Formatted at 2024-05-02 01:55:00 on dist-test-slave-06l8"
I20240502 01:55:00.133142 14446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "382625cd16224c8aa404242500ab3bfa"
format_stamp: "Formatted at 2024-05-02 01:55:00 on dist-test-slave-06l8"
I20240502 01:55:00.138343 14446 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 01:55:00.141674 14460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:00.142956 14446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:55:00.143110 14446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "382625cd16224c8aa404242500ab3bfa"
format_stamp: "Formatted at 2024-05-02 01:55:00 on dist-test-slave-06l8"
I20240502 01:55:00.143304 14446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:00.159261 14446 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:55:00.160084 14446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:00.160353 14446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:00.161288 14446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:00.162739 14446 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:00.162847 14446 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:00.162957 14446 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:00.163025 14446 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:00.193440 14446 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.193:45719
I20240502 01:55:00.193500 14573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.193:45719 every 8 connection(s)
I20240502 01:55:00.194753 14446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:55:00.196131 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 14446
I20240502 01:55:00.196352 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:55:00.199625 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.194:0
--local_ip_for_outbound_sockets=127.31.161.194
--webserver_interface=127.31.161.194
--webserver_port=0
--tserver_master_addrs=127.31.161.254:36883
--builtin_ntp_servers=127.31.161.212:39323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:00.216454 14574 heartbeater.cc:347] Connected to a master server at 127.31.161.254:36883
I20240502 01:55:00.216781 14574 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:00.217758 14574 heartbeater.cc:511] Master 127.31.161.254:36883 requested a full tablet report, sending...
I20240502 01:55:00.265121 14384 ts_manager.cc:194] Registered new tserver with Master: 382625cd16224c8aa404242500ab3bfa (127.31.161.193:45719)
I20240502 01:55:00.266968 14384 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.193:50053
W20240502 01:55:00.543138 14578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:00.543453 14578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:00.543584 14578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:00.551224 14578 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:00.551373 14578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:00.551524 14578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.194
I20240502 01:55:00.560842 14578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:39323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.31.161.194
--webserver_port=0
--tserver_master_addrs=127.31.161.254:36883
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.194
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:00.562220 14578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:00.564034 14578 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:00.575757 14584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:00.576464 14585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:00.576684 14587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:00.576890 14578 server_base.cc:1034] running on GCE node
I20240502 01:55:00.577764 14578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:00.578749 14578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:00.579950 14578 hybrid_clock.cc:648] HybridClock initialized: now 1714614900579899 us; error 45 us; skew 500 ppm
I20240502 01:55:00.580266 14578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:00.582463 14578 webserver.cc:460] Webserver started at http://127.31.161.194:45403/ using document root <none> and password file <none>
I20240502 01:55:00.583221 14578 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:00.583338 14578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:00.583640 14578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:00.586266 14578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c333763da41a4fd8b7b1397b88bff647"
format_stamp: "Formatted at 2024-05-02 01:55:00 on dist-test-slave-06l8"
I20240502 01:55:00.587007 14578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c333763da41a4fd8b7b1397b88bff647"
format_stamp: "Formatted at 2024-05-02 01:55:00 on dist-test-slave-06l8"
I20240502 01:55:00.591936 14578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 01:55:00.595170 14594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:00.596444 14578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:55:00.596587 14578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c333763da41a4fd8b7b1397b88bff647"
format_stamp: "Formatted at 2024-05-02 01:55:00 on dist-test-slave-06l8"
I20240502 01:55:00.596813 14578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:00.616885 14578 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:55:00.617769 14578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:00.618046 14578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:00.619031 14578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:00.620523 14578 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:00.620617 14578 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:00.620725 14578 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:00.620793 14578 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:00.650101 14578 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.194:41573
I20240502 01:55:00.650153 14707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.194:41573 every 8 connection(s)
I20240502 01:55:00.651450 14578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:55:00.654560 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 14578
I20240502 01:55:00.654721 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:55:00.658293 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.195:0
--local_ip_for_outbound_sockets=127.31.161.195
--webserver_interface=127.31.161.195
--webserver_port=0
--tserver_master_addrs=127.31.161.254:36883
--builtin_ntp_servers=127.31.161.212:39323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:00.665160 14708 heartbeater.cc:347] Connected to a master server at 127.31.161.254:36883
I20240502 01:55:00.665513 14708 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:00.666551 14708 heartbeater.cc:511] Master 127.31.161.254:36883 requested a full tablet report, sending...
I20240502 01:55:00.709959 14384 ts_manager.cc:194] Registered new tserver with Master: c333763da41a4fd8b7b1397b88bff647 (127.31.161.194:41573)
I20240502 01:55:00.710906 14384 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.194:47877
W20240502 01:55:00.995126 14711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:00.995425 14711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:00.995558 14711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:01.003239 14711 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:01.003383 14711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:01.003540 14711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.195
I20240502 01:55:01.012529 14711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:39323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.31.161.195
--webserver_port=0
--tserver_master_addrs=127.31.161.254:36883
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.195
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:01.013944 14711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:01.015740 14711 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:01.027511 14718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:01.027513 14720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:01.028131 14717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:01.028380 14711 server_base.cc:1034] running on GCE node
I20240502 01:55:01.028914 14711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:01.029822 14711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:01.031008 14711 hybrid_clock.cc:648] HybridClock initialized: now 1714614901030959 us; error 52 us; skew 500 ppm
I20240502 01:55:01.031301 14711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:01.033406 14711 webserver.cc:460] Webserver started at http://127.31.161.195:45629/ using document root <none> and password file <none>
I20240502 01:55:01.034096 14711 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:01.034204 14711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:01.034498 14711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:01.036967 14711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "43d18c2b045b42029df4e57fd7704436"
format_stamp: "Formatted at 2024-05-02 01:55:01 on dist-test-slave-06l8"
I20240502 01:55:01.037739 14711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "43d18c2b045b42029df4e57fd7704436"
format_stamp: "Formatted at 2024-05-02 01:55:01 on dist-test-slave-06l8"
I20240502 01:55:01.042410 14711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20240502 01:55:01.045498 14726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:01.046736 14711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:55:01.046875 14711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "43d18c2b045b42029df4e57fd7704436"
format_stamp: "Formatted at 2024-05-02 01:55:01 on dist-test-slave-06l8"
I20240502 01:55:01.047058 14711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:01.069255 14711 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:55:01.070112 14711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:01.070384 14711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:01.071322 14711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:01.072793 14711 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:01.072886 14711 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:01.072999 14711 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:01.073065 14711 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:01.102355 14711 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.195:32901
I20240502 01:55:01.102407 14838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.195:32901 every 8 connection(s)
I20240502 01:55:01.103782 14711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:55:01.108431 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 14711
I20240502 01:55:01.108606 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:55:01.112169 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.196:0
--local_ip_for_outbound_sockets=127.31.161.196
--webserver_interface=127.31.161.196
--webserver_port=0
--tserver_master_addrs=127.31.161.254:36883
--builtin_ntp_servers=127.31.161.212:39323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:01.116817 14839 heartbeater.cc:347] Connected to a master server at 127.31.161.254:36883
I20240502 01:55:01.117158 14839 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:01.117969 14839 heartbeater.cc:511] Master 127.31.161.254:36883 requested a full tablet report, sending...
I20240502 01:55:01.162179 14384 ts_manager.cc:194] Registered new tserver with Master: 43d18c2b045b42029df4e57fd7704436 (127.31.161.195:32901)
I20240502 01:55:01.163552 14384 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.195:39941
I20240502 01:55:01.270625 14574 heartbeater.cc:503] Master 127.31.161.254:36883 was elected leader, sending a full tablet report...
W20240502 01:55:01.448273 14842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:01.448550 14842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:01.448688 14842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:01.456393 14842 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:01.456540 14842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:01.456715 14842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.196
I20240502 01:55:01.465973 14842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:39323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.31.161.196
--webserver_port=0
--tserver_master_addrs=127.31.161.254:36883
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.196
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:01.467243 14842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:01.469106 14842 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:01.480638 14848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:01.480823 14851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:01.481444 14849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:01.482024 14842 server_base.cc:1034] running on GCE node
I20240502 01:55:01.482574 14842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:01.483450 14842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:01.484658 14842 hybrid_clock.cc:648] HybridClock initialized: now 1714614901484608 us; error 42 us; skew 500 ppm
I20240502 01:55:01.484957 14842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:01.487025 14842 webserver.cc:460] Webserver started at http://127.31.161.196:33473/ using document root <none> and password file <none>
I20240502 01:55:01.487722 14842 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:01.487830 14842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:01.488121 14842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:01.490631 14842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "bb14db373a304a2bbbeca9f1b6759d1e"
format_stamp: "Formatted at 2024-05-02 01:55:01 on dist-test-slave-06l8"
I20240502 01:55:01.491364 14842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "bb14db373a304a2bbbeca9f1b6759d1e"
format_stamp: "Formatted at 2024-05-02 01:55:01 on dist-test-slave-06l8"
I20240502 01:55:01.496299 14842 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 01:55:01.499648 14857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:01.501008 14842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:55:01.501184 14842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "bb14db373a304a2bbbeca9f1b6759d1e"
format_stamp: "Formatted at 2024-05-02 01:55:01 on dist-test-slave-06l8"
I20240502 01:55:01.501418 14842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:01.516299 14842 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:55:01.517117 14842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:01.517383 14842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:01.518255 14842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:01.519683 14842 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:01.519780 14842 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:01.519888 14842 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:01.519959 14842 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:01.549965 14842 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.196:34533
I20240502 01:55:01.550019 14970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.196:34533 every 8 connection(s)
I20240502 01:55:01.551275 14842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:55:01.552583 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 14842
I20240502 01:55:01.552760 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:55:01.563925 14971 heartbeater.cc:347] Connected to a master server at 127.31.161.254:36883
I20240502 01:55:01.564193 14971 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:01.564952 14971 heartbeater.cc:511] Master 127.31.161.254:36883 requested a full tablet report, sending...
I20240502 01:55:01.608512 14384 ts_manager.cc:194] Registered new tserver with Master: bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196:34533)
I20240502 01:55:01.609493 14384 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.196:41459
I20240502 01:55:01.613391 32391 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 01:55:01.630924 32391 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 01:55:01.631072 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 14578
I20240502 01:55:01.638661 14703 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:01.717269 14708 heartbeater.cc:503] Master 127.31.161.254:36883 was elected leader, sending a full tablet report...
I20240502 01:55:01.718992 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 14578
I20240502 01:55:01.734980 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 14711
I20240502 01:55:01.742861 14834 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:01.821856 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 14711
I20240502 01:55:02.612274 14971 heartbeater.cc:503] Master 127.31.161.254:36883 was elected leader, sending a full tablet report...
I20240502 01:55:05.589336 14384 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42908:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 01:55:05.596071 14384 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240502 01:55:05.625264 14906 tablet_service.cc:1467] Processing CreateTablet for tablet b8f88407a65d4de398128285a8490a78 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1ba208bc11da43d2a765c6489c0daa5d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:05.626893 14906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8f88407a65d4de398128285a8490a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:05.626961 14905 tablet_service.cc:1467] Processing CreateTablet for tablet 6654e083c8e94913aa2a8bd0ccb1af5a (DEFAULT_TABLE table=rebalance_test_table_0 [id=1ba208bc11da43d2a765c6489c0daa5d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:05.627395 14905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6654e083c8e94913aa2a8bd0ccb1af5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:05.628607 14904 tablet_service.cc:1467] Processing CreateTablet for tablet 6d42de8e18d44481b051216dea2a6a55 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1ba208bc11da43d2a765c6489c0daa5d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:05.628365 14508 tablet_service.cc:1467] Processing CreateTablet for tablet b8f88407a65d4de398128285a8490a78 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1ba208bc11da43d2a765c6489c0daa5d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:05.628343 14507 tablet_service.cc:1467] Processing CreateTablet for tablet 6654e083c8e94913aa2a8bd0ccb1af5a (DEFAULT_TABLE table=rebalance_test_table_0 [id=1ba208bc11da43d2a765c6489c0daa5d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:05.629094 14904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d42de8e18d44481b051216dea2a6a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:05.628887 14506 tablet_service.cc:1467] Processing CreateTablet for tablet 6d42de8e18d44481b051216dea2a6a55 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1ba208bc11da43d2a765c6489c0daa5d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:05.629930 14508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8f88407a65d4de398128285a8490a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:05.630398 14507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6654e083c8e94913aa2a8bd0ccb1af5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:05.631057 14506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d42de8e18d44481b051216dea2a6a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:05.643532 14994 tablet_bootstrap.cc:492] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa: Bootstrap starting.
I20240502 01:55:05.644344 14993 tablet_bootstrap.cc:492] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e: Bootstrap starting.
I20240502 01:55:05.647256 14993 tablet_bootstrap.cc:654] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:05.647409 14994 tablet_bootstrap.cc:654] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:05.648360 14993 log.cc:824] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:05.648720 14994 log.cc:824] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:05.652252 14993 tablet_bootstrap.cc:492] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e: No bootstrap required, opened a new log
I20240502 01:55:05.652498 14994 tablet_bootstrap.cc:492] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa: No bootstrap required, opened a new log
I20240502 01:55:05.652501 14993 ts_tablet_manager.cc:1367] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240502 01:55:05.652724 14994 ts_tablet_manager.cc:1367] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20240502 01:55:05.658599 14993 raft_consensus.cc:348] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:05.658907 14994 raft_consensus.cc:348] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:05.659245 14993 raft_consensus.cc:374] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:05.659312 14994 raft_consensus.cc:374] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:05.659363 14993 raft_consensus.cc:729] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb14db373a304a2bbbeca9f1b6759d1e, State: Initialized, Role: FOLLOWER
I20240502 01:55:05.659418 14994 raft_consensus.cc:729] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382625cd16224c8aa404242500ab3bfa, State: Initialized, Role: FOLLOWER
I20240502 01:55:05.659982 14994 consensus_queue.cc:260] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:05.659986 14993 consensus_queue.cc:260] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:05.661681 14993 ts_tablet_manager.cc:1398] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e: Time spent starting tablet: real 0.009s user 0.009s sys 0.001s
I20240502 01:55:05.662043 14993 tablet_bootstrap.cc:492] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e: Bootstrap starting.
I20240502 01:55:05.663398 14993 tablet_bootstrap.cc:654] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:05.663395 14994 ts_tablet_manager.cc:1398] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20240502 01:55:05.663774 14994 tablet_bootstrap.cc:492] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa: Bootstrap starting.
I20240502 01:55:05.665107 14993 tablet_bootstrap.cc:492] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e: No bootstrap required, opened a new log
I20240502 01:55:05.665311 14993 ts_tablet_manager.cc:1367] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240502 01:55:05.665360 14994 tablet_bootstrap.cc:654] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:05.666090 14993 raft_consensus.cc:348] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:05.666335 14993 raft_consensus.cc:374] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:05.666489 14993 raft_consensus.cc:729] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb14db373a304a2bbbeca9f1b6759d1e, State: Initialized, Role: FOLLOWER
I20240502 01:55:05.666862 14993 consensus_queue.cc:260] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:05.667378 14993 ts_tablet_manager.cc:1398] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:55:05.667680 14993 tablet_bootstrap.cc:492] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e: Bootstrap starting.
I20240502 01:55:05.667950 14994 tablet_bootstrap.cc:492] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa: No bootstrap required, opened a new log
I20240502 01:55:05.668102 14994 ts_tablet_manager.cc:1367] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:55:05.669318 14993 tablet_bootstrap.cc:654] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:05.669600 14994 raft_consensus.cc:348] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:05.669836 14994 raft_consensus.cc:374] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:05.669981 14994 raft_consensus.cc:729] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382625cd16224c8aa404242500ab3bfa, State: Initialized, Role: FOLLOWER
I20240502 01:55:05.670347 14994 consensus_queue.cc:260] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:05.670847 14994 ts_tablet_manager.cc:1398] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:55:05.671093 14994 tablet_bootstrap.cc:492] T b8f88407a65d4de398128285a8490a78 P 382625cd16224c8aa404242500ab3bfa: Bootstrap starting.
I20240502 01:55:05.672237 14994 tablet_bootstrap.cc:654] T b8f88407a65d4de398128285a8490a78 P 382625cd16224c8aa404242500ab3bfa: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:05.674134 14994 tablet_bootstrap.cc:492] T b8f88407a65d4de398128285a8490a78 P 382625cd16224c8aa404242500ab3bfa: No bootstrap required, opened a new log
I20240502 01:55:05.674297 14994 ts_tablet_manager.cc:1367] T b8f88407a65d4de398128285a8490a78 P 382625cd16224c8aa404242500ab3bfa: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20240502 01:55:05.674815 14994 raft_consensus.cc:348] T b8f88407a65d4de398128285a8490a78 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:05.674968 14994 raft_consensus.cc:374] T b8f88407a65d4de398128285a8490a78 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:05.675043 14994 raft_consensus.cc:729] T b8f88407a65d4de398128285a8490a78 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382625cd16224c8aa404242500ab3bfa, State: Initialized, Role: FOLLOWER
I20240502 01:55:05.675251 14994 consensus_queue.cc:260] T b8f88407a65d4de398128285a8490a78 P 382625cd16224c8aa404242500ab3bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:05.675633 14994 ts_tablet_manager.cc:1398] T b8f88407a65d4de398128285a8490a78 P 382625cd16224c8aa404242500ab3bfa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:55:05.676543 14993 tablet_bootstrap.cc:492] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e: No bootstrap required, opened a new log
I20240502 01:55:05.676690 14993 ts_tablet_manager.cc:1367] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20240502 01:55:05.677436 14993 raft_consensus.cc:348] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:05.677672 14993 raft_consensus.cc:374] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:05.677762 14993 raft_consensus.cc:729] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb14db373a304a2bbbeca9f1b6759d1e, State: Initialized, Role: FOLLOWER
I20240502 01:55:05.677989 14993 consensus_queue.cc:260] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:05.678450 14993 ts_tablet_manager.cc:1398] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:05.684377 14997 raft_consensus.cc:482] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:05.684597 14997 raft_consensus.cc:504] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:05.686048 14997 leader_election.cc:288] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 382625cd16224c8aa404242500ab3bfa (127.31.161.193:45719)
I20240502 01:55:05.692178 14529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d42de8e18d44481b051216dea2a6a55" candidate_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382625cd16224c8aa404242500ab3bfa" is_pre_election: true
I20240502 01:55:05.692718 14529 raft_consensus.cc:2454] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb14db373a304a2bbbeca9f1b6759d1e in term 0.
I20240502 01:55:05.693339 14860 leader_election.cc:302] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 382625cd16224c8aa404242500ab3bfa, bb14db373a304a2bbbeca9f1b6759d1e; no voters:
I20240502 01:55:05.693706 14997 raft_consensus.cc:2781] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:05.693842 14997 raft_consensus.cc:482] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:05.693930 14997 raft_consensus.cc:3037] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:05.695886 14997 raft_consensus.cc:504] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:05.696360 14997 leader_election.cc:288] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [CANDIDATE]: Term 1 election: Requested vote from peers 382625cd16224c8aa404242500ab3bfa (127.31.161.193:45719)
I20240502 01:55:05.696838 14529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d42de8e18d44481b051216dea2a6a55" candidate_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382625cd16224c8aa404242500ab3bfa"
I20240502 01:55:05.697016 14529 raft_consensus.cc:3037] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:05.699337 14529 raft_consensus.cc:2454] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb14db373a304a2bbbeca9f1b6759d1e in term 1.
I20240502 01:55:05.699697 14860 leader_election.cc:302] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 382625cd16224c8aa404242500ab3bfa, bb14db373a304a2bbbeca9f1b6759d1e; no voters:
I20240502 01:55:05.699959 14997 raft_consensus.cc:2781] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Leader election won for term 1
W20240502 01:55:05.700153 14575 tablet.cc:2369] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:55:05.700412 14997 raft_consensus.cc:686] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [term 1 LEADER]: Becoming Leader. State: Replica: bb14db373a304a2bbbeca9f1b6759d1e, State: Running, Role: LEADER
I20240502 01:55:05.700881 14997 consensus_queue.cc:237] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:05.705401 14384 catalog_manager.cc:5484] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e reported cstate change: term changed from 0 to 1, leader changed from <none> to bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196). New cstate: current_term: 1 leader_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:05.707118 14998 raft_consensus.cc:482] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:05.707273 14998 raft_consensus.cc:504] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:05.707360 14997 raft_consensus.cc:482] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:05.707512 14997 raft_consensus.cc:504] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:05.707928 14997 leader_election.cc:288] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 382625cd16224c8aa404242500ab3bfa (127.31.161.193:45719)
I20240502 01:55:05.708396 14529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8f88407a65d4de398128285a8490a78" candidate_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382625cd16224c8aa404242500ab3bfa" is_pre_election: true
I20240502 01:55:05.708632 14529 raft_consensus.cc:2454] T b8f88407a65d4de398128285a8490a78 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb14db373a304a2bbbeca9f1b6759d1e in term 0.
I20240502 01:55:05.708827 14998 leader_election.cc:288] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196:34533)
I20240502 01:55:05.709153 14860 leader_election.cc:302] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 382625cd16224c8aa404242500ab3bfa, bb14db373a304a2bbbeca9f1b6759d1e; no voters:
I20240502 01:55:05.709460 14997 raft_consensus.cc:2781] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:05.709576 14997 raft_consensus.cc:482] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:05.709707 14997 raft_consensus.cc:3037] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:05.712018 14997 raft_consensus.cc:504] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:05.712527 14997 leader_election.cc:288] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e [CANDIDATE]: Term 1 election: Requested vote from peers 382625cd16224c8aa404242500ab3bfa (127.31.161.193:45719)
I20240502 01:55:05.712973 14529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8f88407a65d4de398128285a8490a78" candidate_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382625cd16224c8aa404242500ab3bfa"
I20240502 01:55:05.713214 14529 raft_consensus.cc:3037] T b8f88407a65d4de398128285a8490a78 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:05.715651 14529 raft_consensus.cc:2454] T b8f88407a65d4de398128285a8490a78 P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb14db373a304a2bbbeca9f1b6759d1e in term 1.
I20240502 01:55:05.716015 14860 leader_election.cc:302] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 382625cd16224c8aa404242500ab3bfa, bb14db373a304a2bbbeca9f1b6759d1e; no voters:
I20240502 01:55:05.716127 14926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6654e083c8e94913aa2a8bd0ccb1af5a" candidate_uuid: "382625cd16224c8aa404242500ab3bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" is_pre_election: true
I20240502 01:55:05.716325 14997 raft_consensus.cc:2781] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:05.716516 14997 raft_consensus.cc:686] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e [term 1 LEADER]: Becoming Leader. State: Replica: bb14db373a304a2bbbeca9f1b6759d1e, State: Running, Role: LEADER
I20240502 01:55:05.716557 14926 raft_consensus.cc:2454] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 382625cd16224c8aa404242500ab3bfa in term 0.
I20240502 01:55:05.716750 14997 consensus_queue.cc:237] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:05.717099 14462 leader_election.cc:302] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 382625cd16224c8aa404242500ab3bfa, bb14db373a304a2bbbeca9f1b6759d1e; no voters:
I20240502 01:55:05.717484 14998 raft_consensus.cc:2781] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:05.717604 14998 raft_consensus.cc:482] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:05.717691 14998 raft_consensus.cc:3037] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:05.719743 14998 raft_consensus.cc:504] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:05.720194 14998 leader_election.cc:288] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [CANDIDATE]: Term 1 election: Requested vote from peers bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196:34533)
I20240502 01:55:05.720655 14926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6654e083c8e94913aa2a8bd0ccb1af5a" candidate_uuid: "382625cd16224c8aa404242500ab3bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb14db373a304a2bbbeca9f1b6759d1e"
I20240502 01:55:05.720842 14926 raft_consensus.cc:3037] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:05.721832 14384 catalog_manager.cc:5484] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e reported cstate change: term changed from 0 to 1, leader changed from <none> to bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196). New cstate: current_term: 1 leader_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:05.722851 14926 raft_consensus.cc:2454] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 382625cd16224c8aa404242500ab3bfa in term 1.
I20240502 01:55:05.723202 14462 leader_election.cc:302] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 382625cd16224c8aa404242500ab3bfa, bb14db373a304a2bbbeca9f1b6759d1e; no voters:
I20240502 01:55:05.723503 14998 raft_consensus.cc:2781] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:05.723942 14998 raft_consensus.cc:686] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [term 1 LEADER]: Becoming Leader. State: Replica: 382625cd16224c8aa404242500ab3bfa, State: Running, Role: LEADER
I20240502 01:55:05.724539 14998 consensus_queue.cc:237] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:05.728847 14384 catalog_manager.cc:5484] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa reported cstate change: term changed from 0 to 1, leader changed from <none> to 382625cd16224c8aa404242500ab3bfa (127.31.161.193). New cstate: current_term: 1 leader_uuid: "382625cd16224c8aa404242500ab3bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } health_report { overall_health: UNKNOWN } } }
W20240502 01:55:05.805945 14972 tablet.cc:2369] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:55:06.074496 15011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:06.074838 15011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:06.082365 15011 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 01:55:06.178455 14998 consensus_queue.cc:1035] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:06.252408 14997 consensus_queue.cc:1035] T b8f88407a65d4de398128285a8490a78 P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:06.270555 14528 raft_consensus.cc:1261] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Refusing update from remote peer bb14db373a304a2bbbeca9f1b6759d1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:06.271368 14997 consensus_queue.cc:1035] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 240.7 ms
time per row: 0.12035 ms
I20240502 01:55:06.465855 14379 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42908:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 01:55:06.466723 14379 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240502 01:55:06.476392 14905 tablet_service.cc:1467] Processing CreateTablet for tablet beb3ef1600054db9903893cd89bd464c (DEFAULT_TABLE table=rebalance_test_table_1 [id=ab8163db47cd4b329cd331b45db9d06c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:06.476532 14506 tablet_service.cc:1467] Processing CreateTablet for tablet beb3ef1600054db9903893cd89bd464c (DEFAULT_TABLE table=rebalance_test_table_1 [id=ab8163db47cd4b329cd331b45db9d06c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:06.476908 14905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb3ef1600054db9903893cd89bd464c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:06.476972 14506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb3ef1600054db9903893cd89bd464c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:06.478374 14904 tablet_service.cc:1467] Processing CreateTablet for tablet 31e69931401143bbb018e06772c15fd8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ab8163db47cd4b329cd331b45db9d06c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:06.478751 14904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31e69931401143bbb018e06772c15fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:06.481491 14906 tablet_service.cc:1467] Processing CreateTablet for tablet 273daa0829624e8bab379a0867748928 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ab8163db47cd4b329cd331b45db9d06c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:06.481876 14906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 273daa0829624e8bab379a0867748928. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:06.485400 14507 tablet_service.cc:1467] Processing CreateTablet for tablet 31e69931401143bbb018e06772c15fd8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ab8163db47cd4b329cd331b45db9d06c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:06.485811 14507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31e69931401143bbb018e06772c15fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:06.486327 14508 tablet_service.cc:1467] Processing CreateTablet for tablet 273daa0829624e8bab379a0867748928 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ab8163db47cd4b329cd331b45db9d06c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:06.492719 14508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 273daa0829624e8bab379a0867748928. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:06.492908 15061 tablet_bootstrap.cc:492] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa: Bootstrap starting.
I20240502 01:55:06.496212 15061 tablet_bootstrap.cc:654] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:06.503206 15060 tablet_bootstrap.cc:492] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e: Bootstrap starting.
I20240502 01:55:06.504519 15061 tablet_bootstrap.cc:492] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa: No bootstrap required, opened a new log
I20240502 01:55:06.504666 15061 ts_tablet_manager.cc:1367] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20240502 01:55:06.505457 15060 tablet_bootstrap.cc:654] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:06.505417 15061 raft_consensus.cc:348] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:06.505690 15061 raft_consensus.cc:374] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:06.505785 15061 raft_consensus.cc:729] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382625cd16224c8aa404242500ab3bfa, State: Initialized, Role: FOLLOWER
I20240502 01:55:06.506012 15061 consensus_queue.cc:260] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:06.506575 15061 ts_tablet_manager.cc:1398] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:06.506974 15061 tablet_bootstrap.cc:492] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa: Bootstrap starting.
I20240502 01:55:06.507635 15060 tablet_bootstrap.cc:492] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e: No bootstrap required, opened a new log
I20240502 01:55:06.507826 15060 ts_tablet_manager.cc:1367] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:55:06.508664 15060 raft_consensus.cc:348] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:06.508891 15060 raft_consensus.cc:374] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:06.509004 15060 raft_consensus.cc:729] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb14db373a304a2bbbeca9f1b6759d1e, State: Initialized, Role: FOLLOWER
I20240502 01:55:06.509307 15060 consensus_queue.cc:260] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:06.510026 15060 ts_tablet_manager.cc:1398] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:55:06.510327 15060 tablet_bootstrap.cc:492] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e: Bootstrap starting.
I20240502 01:55:06.511196 15061 tablet_bootstrap.cc:654] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:06.511869 15060 tablet_bootstrap.cc:654] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:06.519286 15061 tablet_bootstrap.cc:492] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa: No bootstrap required, opened a new log
I20240502 01:55:06.519490 15061 ts_tablet_manager.cc:1367] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20240502 01:55:06.519603 15060 tablet_bootstrap.cc:492] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e: No bootstrap required, opened a new log
I20240502 01:55:06.519769 15060 ts_tablet_manager.cc:1367] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.002s
I20240502 01:55:06.520200 15061 raft_consensus.cc:348] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:06.520383 15061 raft_consensus.cc:374] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:06.520463 15061 raft_consensus.cc:729] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382625cd16224c8aa404242500ab3bfa, State: Initialized, Role: FOLLOWER
I20240502 01:55:06.520537 15060 raft_consensus.cc:348] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:06.520722 15061 consensus_queue.cc:260] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:06.520823 15060 raft_consensus.cc:374] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:06.520892 15060 raft_consensus.cc:729] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb14db373a304a2bbbeca9f1b6759d1e, State: Initialized, Role: FOLLOWER
I20240502 01:55:06.521279 15061 ts_tablet_manager.cc:1398] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:55:06.521592 15061 tablet_bootstrap.cc:492] T 273daa0829624e8bab379a0867748928 P 382625cd16224c8aa404242500ab3bfa: Bootstrap starting.
I20240502 01:55:06.521757 15060 consensus_queue.cc:260] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:06.522924 15061 tablet_bootstrap.cc:654] T 273daa0829624e8bab379a0867748928 P 382625cd16224c8aa404242500ab3bfa: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:06.526278 15060 ts_tablet_manager.cc:1398] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20240502 01:55:06.528079 15061 tablet_bootstrap.cc:492] T 273daa0829624e8bab379a0867748928 P 382625cd16224c8aa404242500ab3bfa: No bootstrap required, opened a new log
I20240502 01:55:06.528231 15061 ts_tablet_manager.cc:1367] T 273daa0829624e8bab379a0867748928 P 382625cd16224c8aa404242500ab3bfa: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240502 01:55:06.528635 15060 tablet_bootstrap.cc:492] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e: Bootstrap starting.
I20240502 01:55:06.528904 15061 raft_consensus.cc:348] T 273daa0829624e8bab379a0867748928 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:06.529067 15061 raft_consensus.cc:374] T 273daa0829624e8bab379a0867748928 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:06.529209 15061 raft_consensus.cc:729] T 273daa0829624e8bab379a0867748928 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382625cd16224c8aa404242500ab3bfa, State: Initialized, Role: FOLLOWER
I20240502 01:55:06.529434 15061 consensus_queue.cc:260] T 273daa0829624e8bab379a0867748928 P 382625cd16224c8aa404242500ab3bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:06.529917 15061 ts_tablet_manager.cc:1398] T 273daa0829624e8bab379a0867748928 P 382625cd16224c8aa404242500ab3bfa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:06.530342 15060 tablet_bootstrap.cc:654] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:06.534154 15060 tablet_bootstrap.cc:492] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e: No bootstrap required, opened a new log
I20240502 01:55:06.534301 15060 ts_tablet_manager.cc:1367] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 01:55:06.534940 15060 raft_consensus.cc:348] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:06.535120 15060 raft_consensus.cc:374] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:06.535243 15060 raft_consensus.cc:729] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb14db373a304a2bbbeca9f1b6759d1e, State: Initialized, Role: FOLLOWER
I20240502 01:55:06.535496 15060 consensus_queue.cc:260] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:06.536034 15060 ts_tablet_manager.cc:1398] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:55:06.545217 15009 raft_consensus.cc:482] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:06.545383 15009 raft_consensus.cc:504] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:06.545958 15009 leader_election.cc:288] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196:34533)
I20240502 01:55:06.546520 14926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb3ef1600054db9903893cd89bd464c" candidate_uuid: "382625cd16224c8aa404242500ab3bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" is_pre_election: true
I20240502 01:55:06.546763 14926 raft_consensus.cc:2454] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 382625cd16224c8aa404242500ab3bfa in term 0.
I20240502 01:55:06.547319 14462 leader_election.cc:302] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 382625cd16224c8aa404242500ab3bfa, bb14db373a304a2bbbeca9f1b6759d1e; no voters:
I20240502 01:55:06.547627 15009 raft_consensus.cc:2781] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:06.547732 15009 raft_consensus.cc:482] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:06.547812 15009 raft_consensus.cc:3037] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:06.550001 15009 raft_consensus.cc:504] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:06.550527 15009 leader_election.cc:288] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [CANDIDATE]: Term 1 election: Requested vote from peers bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196:34533)
I20240502 01:55:06.550998 14926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "beb3ef1600054db9903893cd89bd464c" candidate_uuid: "382625cd16224c8aa404242500ab3bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb14db373a304a2bbbeca9f1b6759d1e"
I20240502 01:55:06.551244 14926 raft_consensus.cc:3037] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:06.554320 14926 raft_consensus.cc:2454] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 382625cd16224c8aa404242500ab3bfa in term 1.
I20240502 01:55:06.554752 14462 leader_election.cc:302] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 382625cd16224c8aa404242500ab3bfa, bb14db373a304a2bbbeca9f1b6759d1e; no voters:
I20240502 01:55:06.555069 15009 raft_consensus.cc:2781] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:06.555207 15009 raft_consensus.cc:686] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [term 1 LEADER]: Becoming Leader. State: Replica: 382625cd16224c8aa404242500ab3bfa, State: Running, Role: LEADER
I20240502 01:55:06.555476 15009 consensus_queue.cc:237] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:06.558288 14379 catalog_manager.cc:5484] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa reported cstate change: term changed from 0 to 1, leader changed from <none> to 382625cd16224c8aa404242500ab3bfa (127.31.161.193). New cstate: current_term: 1 leader_uuid: "382625cd16224c8aa404242500ab3bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:06.728237 15046 raft_consensus.cc:482] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:06.728423 15046 raft_consensus.cc:504] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:06.728972 15046 leader_election.cc:288] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 382625cd16224c8aa404242500ab3bfa (127.31.161.193:45719)
I20240502 01:55:06.729538 14528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "273daa0829624e8bab379a0867748928" candidate_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382625cd16224c8aa404242500ab3bfa" is_pre_election: true
I20240502 01:55:06.729785 14528 raft_consensus.cc:2454] T 273daa0829624e8bab379a0867748928 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb14db373a304a2bbbeca9f1b6759d1e in term 0.
I20240502 01:55:06.730262 14860 leader_election.cc:302] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 382625cd16224c8aa404242500ab3bfa, bb14db373a304a2bbbeca9f1b6759d1e; no voters:
I20240502 01:55:06.730520 15046 raft_consensus.cc:2781] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:06.730630 15046 raft_consensus.cc:482] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:06.730712 15046 raft_consensus.cc:3037] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:06.732632 15046 raft_consensus.cc:504] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:06.733134 15046 leader_election.cc:288] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e [CANDIDATE]: Term 1 election: Requested vote from peers 382625cd16224c8aa404242500ab3bfa (127.31.161.193:45719)
I20240502 01:55:06.733669 14528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "273daa0829624e8bab379a0867748928" candidate_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382625cd16224c8aa404242500ab3bfa"
I20240502 01:55:06.733878 14528 raft_consensus.cc:3037] T 273daa0829624e8bab379a0867748928 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:06.735785 14528 raft_consensus.cc:2454] T 273daa0829624e8bab379a0867748928 P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb14db373a304a2bbbeca9f1b6759d1e in term 1.
I20240502 01:55:06.736164 14860 leader_election.cc:302] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 382625cd16224c8aa404242500ab3bfa, bb14db373a304a2bbbeca9f1b6759d1e; no voters:
I20240502 01:55:06.736451 15046 raft_consensus.cc:2781] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:06.736577 15046 raft_consensus.cc:686] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e [term 1 LEADER]: Becoming Leader. State: Replica: bb14db373a304a2bbbeca9f1b6759d1e, State: Running, Role: LEADER
I20240502 01:55:06.736824 15046 consensus_queue.cc:237] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:06.740145 14379 catalog_manager.cc:5484] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e reported cstate change: term changed from 0 to 1, leader changed from <none> to bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196). New cstate: current_term: 1 leader_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:06.780364 15046 raft_consensus.cc:482] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:06.780599 15046 raft_consensus.cc:504] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:06.781188 15046 leader_election.cc:288] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 382625cd16224c8aa404242500ab3bfa (127.31.161.193:45719)
I20240502 01:55:06.781749 14528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31e69931401143bbb018e06772c15fd8" candidate_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382625cd16224c8aa404242500ab3bfa" is_pre_election: true
I20240502 01:55:06.781950 14528 raft_consensus.cc:2454] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb14db373a304a2bbbeca9f1b6759d1e in term 0.
I20240502 01:55:06.782302 14860 leader_election.cc:302] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 382625cd16224c8aa404242500ab3bfa, bb14db373a304a2bbbeca9f1b6759d1e; no voters:
I20240502 01:55:06.782577 15046 raft_consensus.cc:2781] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:06.782660 15046 raft_consensus.cc:482] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:06.782764 15046 raft_consensus.cc:3037] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:06.784524 15046 raft_consensus.cc:504] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:06.784997 15046 leader_election.cc:288] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [CANDIDATE]: Term 1 election: Requested vote from peers 382625cd16224c8aa404242500ab3bfa (127.31.161.193:45719)
I20240502 01:55:06.785553 14528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31e69931401143bbb018e06772c15fd8" candidate_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382625cd16224c8aa404242500ab3bfa"
I20240502 01:55:06.785748 14528 raft_consensus.cc:3037] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:06.787632 14528 raft_consensus.cc:2454] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb14db373a304a2bbbeca9f1b6759d1e in term 1.
I20240502 01:55:06.787982 14860 leader_election.cc:302] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 382625cd16224c8aa404242500ab3bfa, bb14db373a304a2bbbeca9f1b6759d1e; no voters:
I20240502 01:55:06.788221 15046 raft_consensus.cc:2781] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:06.788376 15046 raft_consensus.cc:686] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [term 1 LEADER]: Becoming Leader. State: Replica: bb14db373a304a2bbbeca9f1b6759d1e, State: Running, Role: LEADER
I20240502 01:55:06.788605 15046 consensus_queue.cc:237] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } }
I20240502 01:55:06.790784 14379 catalog_manager.cc:5484] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e reported cstate change: term changed from 0 to 1, leader changed from <none> to bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196). New cstate: current_term: 1 leader_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:07.136502 15009 consensus_queue.cc:1035] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:07.188896 15071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:07.189242 15071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:07.198531 15071 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 01:55:07.267153 15046 consensus_queue.cc:1035] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:07.324509 14529 raft_consensus.cc:1261] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Refusing update from remote peer bb14db373a304a2bbbeca9f1b6759d1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 01:55:07.325374 15046 consensus_queue.cc:1035] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:07.390614 15074 mvcc.cc:205] Tried to move back new op lower bound from 7023062660385775616 to 7023062658207580160. Current Snapshot: MvccSnapshot[applied={T|T < 7023062660385775616}]
INSERT report
rows total: 2000
time total: 172.52 ms
time per row: 0.0862602 ms
I20240502 01:55:07.515067 14379 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42908:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 01:55:07.515825 14379 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240502 01:55:07.523712 14507 tablet_service.cc:1467] Processing CreateTablet for tablet fc2e73ea09a34079bf7efb148dc99160 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1ca71c9931094ef5af1861989e823dcb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:07.523720 14905 tablet_service.cc:1467] Processing CreateTablet for tablet fc2e73ea09a34079bf7efb148dc99160 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1ca71c9931094ef5af1861989e823dcb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:07.524217 14905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2e73ea09a34079bf7efb148dc99160. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:07.524201 14507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc2e73ea09a34079bf7efb148dc99160. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:07.524504 14906 tablet_service.cc:1467] Processing CreateTablet for tablet 86a26ae98b784f9689fc6f07b0d98dab (DEFAULT_TABLE table=rebalance_test_table_2 [id=1ca71c9931094ef5af1861989e823dcb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:07.524602 14904 tablet_service.cc:1467] Processing CreateTablet for tablet efc65c4da5f34c259006ce1834edac6c (DEFAULT_TABLE table=rebalance_test_table_2 [id=1ca71c9931094ef5af1861989e823dcb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:07.524999 14906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86a26ae98b784f9689fc6f07b0d98dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:07.525514 14904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efc65c4da5f34c259006ce1834edac6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:07.531064 14508 tablet_service.cc:1467] Processing CreateTablet for tablet 86a26ae98b784f9689fc6f07b0d98dab (DEFAULT_TABLE table=rebalance_test_table_2 [id=1ca71c9931094ef5af1861989e823dcb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:07.531473 14508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86a26ae98b784f9689fc6f07b0d98dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:07.532575 15125 tablet_bootstrap.cc:492] T fc2e73ea09a34079bf7efb148dc99160 P bb14db373a304a2bbbeca9f1b6759d1e: Bootstrap starting.
I20240502 01:55:07.534772 15125 tablet_bootstrap.cc:654] T fc2e73ea09a34079bf7efb148dc99160 P bb14db373a304a2bbbeca9f1b6759d1e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:07.534731 14506 tablet_service.cc:1467] Processing CreateTablet for tablet efc65c4da5f34c259006ce1834edac6c (DEFAULT_TABLE table=rebalance_test_table_2 [id=1ca71c9931094ef5af1861989e823dcb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:07.535120 14506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efc65c4da5f34c259006ce1834edac6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:07.535240 15124 tablet_bootstrap.cc:492] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa: Bootstrap starting.
I20240502 01:55:07.537214 15124 tablet_bootstrap.cc:654] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:07.542654 15124 tablet_bootstrap.cc:492] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa: No bootstrap required, opened a new log
I20240502 01:55:07.542785 15124 ts_tablet_manager.cc:1367] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240502 01:55:07.543493 15124 raft_consensus.cc:348] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:07.543673 15125 tablet_bootstrap.cc:492] T fc2e73ea09a34079bf7efb148dc99160 P bb14db373a304a2bbbeca9f1b6759d1e: No bootstrap required, opened a new log
I20240502 01:55:07.543702 15124 raft_consensus.cc:374] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:07.543800 15124 raft_consensus.cc:729] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382625cd16224c8aa404242500ab3bfa, State: Initialized, Role: FOLLOWER
I20240502 01:55:07.543810 15125 ts_tablet_manager.cc:1367] T fc2e73ea09a34079bf7efb148dc99160 P bb14db373a304a2bbbeca9f1b6759d1e: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240502 01:55:07.544029 15124 consensus_queue.cc:260] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:07.544610 15124 ts_tablet_manager.cc:1398] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:55:07.544560 15125 raft_consensus.cc:348] T fc2e73ea09a34079bf7efb148dc99160 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:07.544811 15125 raft_consensus.cc:374] T fc2e73ea09a34079bf7efb148dc99160 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:07.544867 15124 tablet_bootstrap.cc:492] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa: Bootstrap starting.
I20240502 01:55:07.544919 15125 raft_consensus.cc:729] T fc2e73ea09a34079bf7efb148dc99160 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb14db373a304a2bbbeca9f1b6759d1e, State: Initialized, Role: FOLLOWER
I20240502 01:55:07.545202 15125 consensus_queue.cc:260] T fc2e73ea09a34079bf7efb148dc99160 P bb14db373a304a2bbbeca9f1b6759d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:07.545920 15125 ts_tablet_manager.cc:1398] T fc2e73ea09a34079bf7efb148dc99160 P bb14db373a304a2bbbeca9f1b6759d1e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:07.546396 15125 tablet_bootstrap.cc:492] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e: Bootstrap starting.
I20240502 01:55:07.547021 15124 tablet_bootstrap.cc:654] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:07.547989 15125 tablet_bootstrap.cc:654] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:07.550904 15125 tablet_bootstrap.cc:492] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e: No bootstrap required, opened a new log
I20240502 01:55:07.551070 15125 ts_tablet_manager.cc:1367] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240502 01:55:07.551798 15125 raft_consensus.cc:348] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:07.552033 15125 raft_consensus.cc:374] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:07.552125 15125 raft_consensus.cc:729] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb14db373a304a2bbbeca9f1b6759d1e, State: Initialized, Role: FOLLOWER
I20240502 01:55:07.552363 15125 consensus_queue.cc:260] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:07.552886 15125 ts_tablet_manager.cc:1398] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:55:07.553179 15125 tablet_bootstrap.cc:492] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e: Bootstrap starting.
I20240502 01:55:07.554680 15125 tablet_bootstrap.cc:654] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:07.556633 15125 tablet_bootstrap.cc:492] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e: No bootstrap required, opened a new log
I20240502 01:55:07.556766 15125 ts_tablet_manager.cc:1367] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:55:07.557431 15125 raft_consensus.cc:348] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:07.557585 15125 raft_consensus.cc:374] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:07.557679 15125 raft_consensus.cc:729] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb14db373a304a2bbbeca9f1b6759d1e, State: Initialized, Role: FOLLOWER
I20240502 01:55:07.557911 15125 consensus_queue.cc:260] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:07.558393 15125 ts_tablet_manager.cc:1398] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:55:07.561923 15124 tablet_bootstrap.cc:492] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa: No bootstrap required, opened a new log
I20240502 01:55:07.562173 15124 ts_tablet_manager.cc:1367] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20240502 01:55:07.562780 15124 raft_consensus.cc:348] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:07.562960 15124 raft_consensus.cc:374] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:07.563028 15124 raft_consensus.cc:729] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382625cd16224c8aa404242500ab3bfa, State: Initialized, Role: FOLLOWER
I20240502 01:55:07.563261 15124 consensus_queue.cc:260] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:07.564348 15124 ts_tablet_manager.cc:1398] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:07.564604 15124 tablet_bootstrap.cc:492] T 86a26ae98b784f9689fc6f07b0d98dab P 382625cd16224c8aa404242500ab3bfa: Bootstrap starting.
I20240502 01:55:07.566020 15124 tablet_bootstrap.cc:654] T 86a26ae98b784f9689fc6f07b0d98dab P 382625cd16224c8aa404242500ab3bfa: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:07.574090 15124 tablet_bootstrap.cc:492] T 86a26ae98b784f9689fc6f07b0d98dab P 382625cd16224c8aa404242500ab3bfa: No bootstrap required, opened a new log
I20240502 01:55:07.574249 15124 ts_tablet_manager.cc:1367] T 86a26ae98b784f9689fc6f07b0d98dab P 382625cd16224c8aa404242500ab3bfa: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20240502 01:55:07.574810 15124 raft_consensus.cc:348] T 86a26ae98b784f9689fc6f07b0d98dab P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:07.574961 15124 raft_consensus.cc:374] T 86a26ae98b784f9689fc6f07b0d98dab P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:07.575045 15124 raft_consensus.cc:729] T 86a26ae98b784f9689fc6f07b0d98dab P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 382625cd16224c8aa404242500ab3bfa, State: Initialized, Role: FOLLOWER
I20240502 01:55:07.575264 15124 consensus_queue.cc:260] T 86a26ae98b784f9689fc6f07b0d98dab P 382625cd16224c8aa404242500ab3bfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:07.575707 15124 ts_tablet_manager.cc:1398] T 86a26ae98b784f9689fc6f07b0d98dab P 382625cd16224c8aa404242500ab3bfa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:55:07.674868 15009 raft_consensus.cc:482] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:07.675073 15009 raft_consensus.cc:504] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:07.675606 15009 leader_election.cc:288] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196:34533)
I20240502 01:55:07.676246 14926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc65c4da5f34c259006ce1834edac6c" candidate_uuid: "382625cd16224c8aa404242500ab3bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" is_pre_election: true
I20240502 01:55:07.676465 14926 raft_consensus.cc:2454] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 382625cd16224c8aa404242500ab3bfa in term 0.
I20240502 01:55:07.676928 14462 leader_election.cc:302] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 382625cd16224c8aa404242500ab3bfa, bb14db373a304a2bbbeca9f1b6759d1e; no voters:
I20240502 01:55:07.677294 15009 raft_consensus.cc:2781] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:07.677423 15009 raft_consensus.cc:482] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:07.677505 15009 raft_consensus.cc:3037] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:07.679359 15009 raft_consensus.cc:504] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:07.679797 15009 leader_election.cc:288] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [CANDIDATE]: Term 1 election: Requested vote from peers bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196:34533)
I20240502 01:55:07.680369 14926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efc65c4da5f34c259006ce1834edac6c" candidate_uuid: "382625cd16224c8aa404242500ab3bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb14db373a304a2bbbeca9f1b6759d1e"
I20240502 01:55:07.680591 14926 raft_consensus.cc:3037] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:07.682601 14926 raft_consensus.cc:2454] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 382625cd16224c8aa404242500ab3bfa in term 1.
I20240502 01:55:07.682977 14462 leader_election.cc:302] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 382625cd16224c8aa404242500ab3bfa, bb14db373a304a2bbbeca9f1b6759d1e; no voters:
I20240502 01:55:07.683252 15009 raft_consensus.cc:2781] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:07.683389 15009 raft_consensus.cc:686] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [term 1 LEADER]: Becoming Leader. State: Replica: 382625cd16224c8aa404242500ab3bfa, State: Running, Role: LEADER
I20240502 01:55:07.683653 15009 consensus_queue.cc:237] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:07.684558 15073 raft_consensus.cc:482] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:07.684728 15073 raft_consensus.cc:504] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:07.685374 15073 leader_election.cc:288] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196:34533)
I20240502 01:55:07.685815 14926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2e73ea09a34079bf7efb148dc99160" candidate_uuid: "382625cd16224c8aa404242500ab3bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" is_pre_election: true
I20240502 01:55:07.686057 14926 raft_consensus.cc:2454] T fc2e73ea09a34079bf7efb148dc99160 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 382625cd16224c8aa404242500ab3bfa in term 0.
I20240502 01:55:07.686434 14462 leader_election.cc:302] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 382625cd16224c8aa404242500ab3bfa, bb14db373a304a2bbbeca9f1b6759d1e; no voters:
I20240502 01:55:07.686216 14384 catalog_manager.cc:5484] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa reported cstate change: term changed from 0 to 1, leader changed from <none> to 382625cd16224c8aa404242500ab3bfa (127.31.161.193). New cstate: current_term: 1 leader_uuid: "382625cd16224c8aa404242500ab3bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:07.686704 15073 raft_consensus.cc:2781] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:07.686812 15073 raft_consensus.cc:482] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:07.686904 15073 raft_consensus.cc:3037] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:07.689178 15073 raft_consensus.cc:504] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:07.689612 15073 leader_election.cc:288] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa [CANDIDATE]: Term 1 election: Requested vote from peers bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196:34533)
I20240502 01:55:07.690047 14926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc2e73ea09a34079bf7efb148dc99160" candidate_uuid: "382625cd16224c8aa404242500ab3bfa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb14db373a304a2bbbeca9f1b6759d1e"
I20240502 01:55:07.690227 14926 raft_consensus.cc:3037] T fc2e73ea09a34079bf7efb148dc99160 P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:07.691991 14926 raft_consensus.cc:2454] T fc2e73ea09a34079bf7efb148dc99160 P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 382625cd16224c8aa404242500ab3bfa in term 1.
I20240502 01:55:07.692328 14462 leader_election.cc:302] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 382625cd16224c8aa404242500ab3bfa, bb14db373a304a2bbbeca9f1b6759d1e; no voters:
I20240502 01:55:07.692600 15073 raft_consensus.cc:2781] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:07.692732 15073 raft_consensus.cc:686] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa [term 1 LEADER]: Becoming Leader. State: Replica: 382625cd16224c8aa404242500ab3bfa, State: Running, Role: LEADER
I20240502 01:55:07.692951 15073 consensus_queue.cc:237] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:07.695322 14384 catalog_manager.cc:5484] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa reported cstate change: term changed from 0 to 1, leader changed from <none> to 382625cd16224c8aa404242500ab3bfa (127.31.161.193). New cstate: current_term: 1 leader_uuid: "382625cd16224c8aa404242500ab3bfa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:07.764685 15114 raft_consensus.cc:482] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:07.764864 15114 raft_consensus.cc:504] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:07.765424 15114 leader_election.cc:288] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 382625cd16224c8aa404242500ab3bfa (127.31.161.193:45719)
I20240502 01:55:07.765904 14529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86a26ae98b784f9689fc6f07b0d98dab" candidate_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382625cd16224c8aa404242500ab3bfa" is_pre_election: true
I20240502 01:55:07.766144 14529 raft_consensus.cc:2454] T 86a26ae98b784f9689fc6f07b0d98dab P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb14db373a304a2bbbeca9f1b6759d1e in term 0.
I20240502 01:55:07.766577 14860 leader_election.cc:302] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 382625cd16224c8aa404242500ab3bfa, bb14db373a304a2bbbeca9f1b6759d1e; no voters:
I20240502 01:55:07.766844 15114 raft_consensus.cc:2781] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:07.766948 15114 raft_consensus.cc:482] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:07.767028 15114 raft_consensus.cc:3037] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:07.768893 15114 raft_consensus.cc:504] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:07.769320 15114 leader_election.cc:288] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [CANDIDATE]: Term 1 election: Requested vote from peers 382625cd16224c8aa404242500ab3bfa (127.31.161.193:45719)
I20240502 01:55:07.769731 14529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86a26ae98b784f9689fc6f07b0d98dab" candidate_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "382625cd16224c8aa404242500ab3bfa"
I20240502 01:55:07.769920 14529 raft_consensus.cc:3037] T 86a26ae98b784f9689fc6f07b0d98dab P 382625cd16224c8aa404242500ab3bfa [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:07.771734 14529 raft_consensus.cc:2454] T 86a26ae98b784f9689fc6f07b0d98dab P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb14db373a304a2bbbeca9f1b6759d1e in term 1.
I20240502 01:55:07.772091 14860 leader_election.cc:302] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 382625cd16224c8aa404242500ab3bfa, bb14db373a304a2bbbeca9f1b6759d1e; no voters:
I20240502 01:55:07.772357 15114 raft_consensus.cc:2781] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:07.772481 15114 raft_consensus.cc:686] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [term 1 LEADER]: Becoming Leader. State: Replica: bb14db373a304a2bbbeca9f1b6759d1e, State: Running, Role: LEADER
I20240502 01:55:07.772692 15114 consensus_queue.cc:237] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } }
I20240502 01:55:07.775302 14384 catalog_manager.cc:5484] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e reported cstate change: term changed from 0 to 1, leader changed from <none> to bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196). New cstate: current_term: 1 leader_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:08.141136 15073 consensus_queue.cc:1035] T fc2e73ea09a34079bf7efb148dc99160 P 382625cd16224c8aa404242500ab3bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:08.153112 15135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:08.153429 15135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:08.160885 15135 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:55:08.168434 15114 consensus_queue.cc:1035] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:08.176041 15009 consensus_queue.cc:1035] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 146.972 ms
time per row: 0.0734861 ms
I20240502 01:55:08.450989 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.194:41573
--local_ip_for_outbound_sockets=127.31.161.194
--tserver_master_addrs=127.31.161.254:36883
--webserver_port=45403
--webserver_interface=127.31.161.194
--builtin_ntp_servers=127.31.161.212:39323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:55:08.786315 15184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:08.786581 15184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:08.786705 15184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:08.793936 15184 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:08.794059 15184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:08.794211 15184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.194
I20240502 01:55:08.803073 15184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:39323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.194:41573
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.31.161.194
--webserver_port=45403
--tserver_master_addrs=127.31.161.254:36883
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.194
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:08.804392 15184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:08.806591 15184 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:08.818485 15189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:08.818723 15192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:08.819412 15190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:08.820094 15184 server_base.cc:1034] running on GCE node
I20240502 01:55:08.820633 15184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:08.821524 15184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:08.822716 15184 hybrid_clock.cc:648] HybridClock initialized: now 1714614908822669 us; error 47 us; skew 500 ppm
I20240502 01:55:08.823007 15184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:08.825016 15184 webserver.cc:460] Webserver started at http://127.31.161.194:45403/ using document root <none> and password file <none>
I20240502 01:55:08.825722 15184 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:08.825845 15184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:08.830830 15184 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 01:55:08.833563 15198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:08.834838 15184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 01:55:08.834991 15184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c333763da41a4fd8b7b1397b88bff647"
format_stamp: "Formatted at 2024-05-02 01:55:00 on dist-test-slave-06l8"
I20240502 01:55:08.835765 15184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:08.855338 15184 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:55:08.856112 15184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:08.856353 15184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:08.857303 15184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:08.858768 15184 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:08.858839 15184 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:08.858952 15184 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:08.859000 15184 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:08.888054 15184 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.194:41573
I20240502 01:55:08.888096 15311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.194:41573 every 8 connection(s)
I20240502 01:55:08.889475 15184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:55:08.890727 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 15184
I20240502 01:55:08.891322 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.195:32901
--local_ip_for_outbound_sockets=127.31.161.195
--tserver_master_addrs=127.31.161.254:36883
--webserver_port=45629
--webserver_interface=127.31.161.195
--builtin_ntp_servers=127.31.161.212:39323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:08.902055 15312 heartbeater.cc:347] Connected to a master server at 127.31.161.254:36883
I20240502 01:55:08.902359 15312 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:08.903362 15312 heartbeater.cc:511] Master 127.31.161.254:36883 requested a full tablet report, sending...
I20240502 01:55:08.905164 14384 ts_manager.cc:194] Re-registered known tserver with Master: c333763da41a4fd8b7b1397b88bff647 (127.31.161.194:41573)
I20240502 01:55:08.906245 14384 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.194:43009
W20240502 01:55:09.233768 15316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:09.234062 15316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:09.234215 15316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:09.244241 15316 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:09.244375 15316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:09.244535 15316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.195
I20240502 01:55:09.253943 15316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:39323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.195:32901
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.31.161.195
--webserver_port=45629
--tserver_master_addrs=127.31.161.254:36883
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.195
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:09.255242 15316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:09.257258 15316 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:09.268466 15323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:09.269140 15325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:09.269383 15316 server_base.cc:1034] running on GCE node
W20240502 01:55:09.269455 15322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:09.270018 15316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:09.270881 15316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:09.272081 15316 hybrid_clock.cc:648] HybridClock initialized: now 1714614909272028 us; error 43 us; skew 500 ppm
I20240502 01:55:09.272393 15316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:09.274490 15316 webserver.cc:460] Webserver started at http://127.31.161.195:45629/ using document root <none> and password file <none>
I20240502 01:55:09.275189 15316 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:09.275319 15316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:09.280584 15316 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 01:55:09.283191 15331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:09.284441 15316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:55:09.284602 15316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "43d18c2b045b42029df4e57fd7704436"
format_stamp: "Formatted at 2024-05-02 01:55:01 on dist-test-slave-06l8"
I20240502 01:55:09.285426 15316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:09.303720 15316 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:55:09.304519 15316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:09.304816 15316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:09.305881 15316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:09.307396 15316 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:09.307474 15316 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:09.307598 15316 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:09.307651 15316 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:09.337284 15316 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.195:32901
I20240502 01:55:09.337352 15444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.195:32901 every 8 connection(s)
I20240502 01:55:09.338754 15316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:55:09.342738 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 15316
I20240502 01:55:09.353296 15445 heartbeater.cc:347] Connected to a master server at 127.31.161.254:36883
I20240502 01:55:09.353615 15445 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:09.354614 15445 heartbeater.cc:511] Master 127.31.161.254:36883 requested a full tablet report, sending...
I20240502 01:55:09.356088 14379 ts_manager.cc:194] Re-registered known tserver with Master: 43d18c2b045b42029df4e57fd7704436 (127.31.161.195:32901)
I20240502 01:55:09.356936 14379 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.195:59621
I20240502 01:55:09.834698 14508 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:09.835212 14905 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:09.850196 15247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:09.851006 15379 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:09.909018 15312 heartbeater.cc:503] Master 127.31.161.254:36883 was elected leader, sending a full tablet report...
I20240502 01:55:10.003496 14508 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:10.012228 15379 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:10.012251 14905 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:10.024103 15247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:10.191411 15379 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:10.191650 14508 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:10.203100 14905 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:10.206003 15247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:10.344714 14508 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:10.347349 14905 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:10.355086 15379 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:10.359792 15445 heartbeater.cc:503] Master 127.31.161.254:36883 was elected leader, sending a full tablet report...
I20240502 01:55:10.360441 15247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:10.423421 14925 consensus_queue.cc:237] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: true } }
I20240502 01:55:10.425777 14529 raft_consensus.cc:1261] T 86a26ae98b784f9689fc6f07b0d98dab P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Refusing update from remote peer bb14db373a304a2bbbeca9f1b6759d1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:10.426352 15114 consensus_queue.cc:1035] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:10.429244 15585 raft_consensus.cc:2932] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 43d18c2b045b42029df4e57fd7704436 (127.31.161.195) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: true } } }
I20240502 01:55:10.429914 14529 raft_consensus.cc:2932] T 86a26ae98b784f9689fc6f07b0d98dab P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 43d18c2b045b42029df4e57fd7704436 (127.31.161.195) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: true } } }
W20240502 01:55:10.432957 14861 consensus_peers.cc:485] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e -> Peer 43d18c2b045b42029df4e57fd7704436 (127.31.161.195:32901): Couldn't send request to peer 43d18c2b045b42029df4e57fd7704436. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 86a26ae98b784f9689fc6f07b0d98dab. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:10.435786 14384 catalog_manager.cc:5484] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e reported cstate change: config changed from index -1 to 4, NON_VOTER 43d18c2b045b42029df4e57fd7704436 (127.31.161.195) added. New cstate: current_term: 1 leader_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:10.446835 14529 consensus_queue.cc:237] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: true } }
I20240502 01:55:10.449335 14925 raft_consensus.cc:1261] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Refusing update from remote peer 382625cd16224c8aa404242500ab3bfa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:10.449913 15009 consensus_queue.cc:1035] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:10.453243 14925 raft_consensus.cc:2932] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c333763da41a4fd8b7b1397b88bff647 (127.31.161.194) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: true } } }
W20240502 01:55:10.455633 14464 consensus_peers.cc:485] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa -> Peer c333763da41a4fd8b7b1397b88bff647 (127.31.161.194:41573): Couldn't send request to peer c333763da41a4fd8b7b1397b88bff647. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: beb3ef1600054db9903893cd89bd464c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:10.452513 15599 raft_consensus.cc:2932] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c333763da41a4fd8b7b1397b88bff647 (127.31.161.194) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: true } } }
I20240502 01:55:10.457641 14379 catalog_manager.cc:5484] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e reported cstate change: config changed from index -1 to 4, NON_VOTER c333763da41a4fd8b7b1397b88bff647 (127.31.161.194) added. New cstate: current_term: 1 leader_uuid: "382625cd16224c8aa404242500ab3bfa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: true } } }
I20240502 01:55:10.471221 14925 consensus_queue.cc:237] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: true } }
I20240502 01:55:10.473054 14529 raft_consensus.cc:1261] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Refusing update from remote peer bb14db373a304a2bbbeca9f1b6759d1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:10.473268 14861 consensus_peers.cc:485] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e -> Peer 43d18c2b045b42029df4e57fd7704436 (127.31.161.195:32901): Couldn't send request to peer 43d18c2b045b42029df4e57fd7704436. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6d42de8e18d44481b051216dea2a6a55. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:10.473574 15114 consensus_queue.cc:1035] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:10.475823 15585 raft_consensus.cc:2932] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 43d18c2b045b42029df4e57fd7704436 (127.31.161.195) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: true } } }
I20240502 01:55:10.476399 14529 raft_consensus.cc:2932] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 43d18c2b045b42029df4e57fd7704436 (127.31.161.195) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: true } } }
I20240502 01:55:10.479987 14384 catalog_manager.cc:5484] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e reported cstate change: config changed from index -1 to 4, NON_VOTER 43d18c2b045b42029df4e57fd7704436 (127.31.161.195) added. New cstate: current_term: 1 leader_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:10.490927 14529 consensus_queue.cc:237] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: true } }
I20240502 01:55:10.492789 14925 raft_consensus.cc:1261] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Refusing update from remote peer 382625cd16224c8aa404242500ab3bfa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:10.492990 14464 consensus_peers.cc:485] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa -> Peer c333763da41a4fd8b7b1397b88bff647 (127.31.161.194:41573): Couldn't send request to peer c333763da41a4fd8b7b1397b88bff647. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6654e083c8e94913aa2a8bd0ccb1af5a. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:10.493325 15599 consensus_queue.cc:1035] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:10.495810 15009 raft_consensus.cc:2932] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c333763da41a4fd8b7b1397b88bff647 (127.31.161.194) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: true } } }
I20240502 01:55:10.496438 14925 raft_consensus.cc:2932] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c333763da41a4fd8b7b1397b88bff647 (127.31.161.194) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: true } } }
I20240502 01:55:10.499975 14379 catalog_manager.cc:5484] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa reported cstate change: config changed from index -1 to 4, NON_VOTER c333763da41a4fd8b7b1397b88bff647 (127.31.161.194) added. New cstate: current_term: 1 leader_uuid: "382625cd16224c8aa404242500ab3bfa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:10.512148 14925 consensus_queue.cc:237] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: true } }
I20240502 01:55:10.513972 14529 raft_consensus.cc:1261] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Refusing update from remote peer bb14db373a304a2bbbeca9f1b6759d1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:10.514148 14861 consensus_peers.cc:485] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e -> Peer 43d18c2b045b42029df4e57fd7704436 (127.31.161.195:32901): Couldn't send request to peer 43d18c2b045b42029df4e57fd7704436. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31e69931401143bbb018e06772c15fd8. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:10.514505 15114 consensus_queue.cc:1035] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:10.516834 15585 raft_consensus.cc:2932] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 43d18c2b045b42029df4e57fd7704436 (127.31.161.195) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: true } } }
I20240502 01:55:10.517769 14529 raft_consensus.cc:2932] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 43d18c2b045b42029df4e57fd7704436 (127.31.161.195) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: true } } }
I20240502 01:55:10.522166 14384 catalog_manager.cc:5484] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e reported cstate change: config changed from index -1 to 4, NON_VOTER 43d18c2b045b42029df4e57fd7704436 (127.31.161.195) added. New cstate: current_term: 1 leader_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:10.532603 14529 consensus_queue.cc:237] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: true } }
I20240502 01:55:10.534651 14925 raft_consensus.cc:1261] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Refusing update from remote peer 382625cd16224c8aa404242500ab3bfa: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 01:55:10.534868 14464 consensus_peers.cc:485] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa -> Peer c333763da41a4fd8b7b1397b88bff647 (127.31.161.194:41573): Couldn't send request to peer c333763da41a4fd8b7b1397b88bff647. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: efc65c4da5f34c259006ce1834edac6c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:10.535208 15599 consensus_queue.cc:1035] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:10.537590 15009 raft_consensus.cc:2932] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c333763da41a4fd8b7b1397b88bff647 (127.31.161.194) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: true } } }
I20240502 01:55:10.538192 14925 raft_consensus.cc:2932] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c333763da41a4fd8b7b1397b88bff647 (127.31.161.194) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: true } } }
I20240502 01:55:10.542690 14379 catalog_manager.cc:5484] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa reported cstate change: config changed from index -1 to 4, NON_VOTER c333763da41a4fd8b7b1397b88bff647 (127.31.161.194) added. New cstate: current_term: 1 leader_uuid: "382625cd16224c8aa404242500ab3bfa" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:10.891259 15691 ts_tablet_manager.cc:902] T 31e69931401143bbb018e06772c15fd8 P 43d18c2b045b42029df4e57fd7704436: Initiating tablet copy from peer bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196:34533)
I20240502 01:55:10.893222 15691 tablet_copy_client.cc:308] T 31e69931401143bbb018e06772c15fd8 P 43d18c2b045b42029df4e57fd7704436: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:34533
I20240502 01:55:10.899669 14946 tablet_copy_service.cc:140] P bb14db373a304a2bbbeca9f1b6759d1e: Received BeginTabletCopySession request for tablet 31e69931401143bbb018e06772c15fd8 from peer 43d18c2b045b42029df4e57fd7704436 ({username='slave'} at 127.31.161.195:54335)
I20240502 01:55:10.899901 14946 tablet_copy_service.cc:161] P bb14db373a304a2bbbeca9f1b6759d1e: Beginning new tablet copy session on tablet 31e69931401143bbb018e06772c15fd8 from peer 43d18c2b045b42029df4e57fd7704436 at {username='slave'} at 127.31.161.195:54335: session id = 43d18c2b045b42029df4e57fd7704436-31e69931401143bbb018e06772c15fd8
I20240502 01:55:10.902558 14946 tablet_copy_source_session.cc:205] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:10.906965 15691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31e69931401143bbb018e06772c15fd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:10.916728 15691 tablet_copy_client.cc:783] T 31e69931401143bbb018e06772c15fd8 P 43d18c2b045b42029df4e57fd7704436: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:10.917038 15691 tablet_copy_client.cc:647] T 31e69931401143bbb018e06772c15fd8 P 43d18c2b045b42029df4e57fd7704436: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:10.919268 15691 tablet_copy_client.cc:523] T 31e69931401143bbb018e06772c15fd8 P 43d18c2b045b42029df4e57fd7704436: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:10.922127 15691 tablet_bootstrap.cc:492] T 31e69931401143bbb018e06772c15fd8 P 43d18c2b045b42029df4e57fd7704436: Bootstrap starting.
I20240502 01:55:10.930469 15691 log.cc:824] T 31e69931401143bbb018e06772c15fd8 P 43d18c2b045b42029df4e57fd7704436: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:10.950891 15698 ts_tablet_manager.cc:902] T 86a26ae98b784f9689fc6f07b0d98dab P 43d18c2b045b42029df4e57fd7704436: Initiating tablet copy from peer bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196:34533)
I20240502 01:55:10.953445 15698 tablet_copy_client.cc:308] T 86a26ae98b784f9689fc6f07b0d98dab P 43d18c2b045b42029df4e57fd7704436: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:34533
I20240502 01:55:10.954172 14946 tablet_copy_service.cc:140] P bb14db373a304a2bbbeca9f1b6759d1e: Received BeginTabletCopySession request for tablet 86a26ae98b784f9689fc6f07b0d98dab from peer 43d18c2b045b42029df4e57fd7704436 ({username='slave'} at 127.31.161.195:54335)
I20240502 01:55:10.954360 14946 tablet_copy_service.cc:161] P bb14db373a304a2bbbeca9f1b6759d1e: Beginning new tablet copy session on tablet 86a26ae98b784f9689fc6f07b0d98dab from peer 43d18c2b045b42029df4e57fd7704436 at {username='slave'} at 127.31.161.195:54335: session id = 43d18c2b045b42029df4e57fd7704436-86a26ae98b784f9689fc6f07b0d98dab
I20240502 01:55:10.956645 14946 tablet_copy_source_session.cc:205] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:10.958115 15698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86a26ae98b784f9689fc6f07b0d98dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:10.964082 15691 tablet_bootstrap.cc:492] T 31e69931401143bbb018e06772c15fd8 P 43d18c2b045b42029df4e57fd7704436: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:10.964702 15691 tablet_bootstrap.cc:492] T 31e69931401143bbb018e06772c15fd8 P 43d18c2b045b42029df4e57fd7704436: Bootstrap complete.
I20240502 01:55:10.964730 15698 tablet_copy_client.cc:783] T 86a26ae98b784f9689fc6f07b0d98dab P 43d18c2b045b42029df4e57fd7704436: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:10.965551 15698 tablet_copy_client.cc:647] T 86a26ae98b784f9689fc6f07b0d98dab P 43d18c2b045b42029df4e57fd7704436: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:10.965719 15691 ts_tablet_manager.cc:1367] T 31e69931401143bbb018e06772c15fd8 P 43d18c2b045b42029df4e57fd7704436: Time spent bootstrapping tablet: real 0.044s user 0.037s sys 0.006s
I20240502 01:55:10.967497 15698 tablet_copy_client.cc:523] T 86a26ae98b784f9689fc6f07b0d98dab P 43d18c2b045b42029df4e57fd7704436: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:10.971130 15698 tablet_bootstrap.cc:492] T 86a26ae98b784f9689fc6f07b0d98dab P 43d18c2b045b42029df4e57fd7704436: Bootstrap starting.
I20240502 01:55:10.972194 15691 raft_consensus.cc:348] T 31e69931401143bbb018e06772c15fd8 P 43d18c2b045b42029df4e57fd7704436 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: true } }
I20240502 01:55:10.972682 15691 raft_consensus.cc:729] T 31e69931401143bbb018e06772c15fd8 P 43d18c2b045b42029df4e57fd7704436 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 43d18c2b045b42029df4e57fd7704436, State: Initialized, Role: LEARNER
I20240502 01:55:10.973325 15691 consensus_queue.cc:260] T 31e69931401143bbb018e06772c15fd8 P 43d18c2b045b42029df4e57fd7704436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: true } }
I20240502 01:55:10.975208 15691 ts_tablet_manager.cc:1398] T 31e69931401143bbb018e06772c15fd8 P 43d18c2b045b42029df4e57fd7704436: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20240502 01:55:10.976137 14946 tablet_copy_service.cc:342] P bb14db373a304a2bbbeca9f1b6759d1e: Request end of tablet copy session 43d18c2b045b42029df4e57fd7704436-31e69931401143bbb018e06772c15fd8 received from {username='slave'} at 127.31.161.195:54335
I20240502 01:55:10.976316 14946 tablet_copy_service.cc:433] P bb14db373a304a2bbbeca9f1b6759d1e: ending tablet copy session 43d18c2b045b42029df4e57fd7704436-31e69931401143bbb018e06772c15fd8 on tablet 31e69931401143bbb018e06772c15fd8 with peer 43d18c2b045b42029df4e57fd7704436
I20240502 01:55:10.979804 15705 ts_tablet_manager.cc:902] T efc65c4da5f34c259006ce1834edac6c P c333763da41a4fd8b7b1397b88bff647: Initiating tablet copy from peer 382625cd16224c8aa404242500ab3bfa (127.31.161.193:45719)
I20240502 01:55:10.981606 15705 tablet_copy_client.cc:308] T efc65c4da5f34c259006ce1834edac6c P c333763da41a4fd8b7b1397b88bff647: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.193:45719
I20240502 01:55:10.991621 14549 tablet_copy_service.cc:140] P 382625cd16224c8aa404242500ab3bfa: Received BeginTabletCopySession request for tablet efc65c4da5f34c259006ce1834edac6c from peer c333763da41a4fd8b7b1397b88bff647 ({username='slave'} at 127.31.161.194:45553)
I20240502 01:55:10.991884 14549 tablet_copy_service.cc:161] P 382625cd16224c8aa404242500ab3bfa: Beginning new tablet copy session on tablet efc65c4da5f34c259006ce1834edac6c from peer c333763da41a4fd8b7b1397b88bff647 at {username='slave'} at 127.31.161.194:45553: session id = c333763da41a4fd8b7b1397b88bff647-efc65c4da5f34c259006ce1834edac6c
I20240502 01:55:10.995616 14549 tablet_copy_source_session.cc:205] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:11.000695 15705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efc65c4da5f34c259006ce1834edac6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:11.007210 15698 tablet_bootstrap.cc:492] T 86a26ae98b784f9689fc6f07b0d98dab P 43d18c2b045b42029df4e57fd7704436: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:11.007577 15698 tablet_bootstrap.cc:492] T 86a26ae98b784f9689fc6f07b0d98dab P 43d18c2b045b42029df4e57fd7704436: Bootstrap complete.
I20240502 01:55:11.007822 15698 ts_tablet_manager.cc:1367] T 86a26ae98b784f9689fc6f07b0d98dab P 43d18c2b045b42029df4e57fd7704436: Time spent bootstrapping tablet: real 0.037s user 0.032s sys 0.004s
I20240502 01:55:11.008569 15698 raft_consensus.cc:348] T 86a26ae98b784f9689fc6f07b0d98dab P 43d18c2b045b42029df4e57fd7704436 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: true } }
I20240502 01:55:11.008766 15698 raft_consensus.cc:729] T 86a26ae98b784f9689fc6f07b0d98dab P 43d18c2b045b42029df4e57fd7704436 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 43d18c2b045b42029df4e57fd7704436, State: Initialized, Role: LEARNER
I20240502 01:55:11.008965 15698 consensus_queue.cc:260] T 86a26ae98b784f9689fc6f07b0d98dab P 43d18c2b045b42029df4e57fd7704436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: true } }
I20240502 01:55:11.009457 15698 ts_tablet_manager.cc:1398] T 86a26ae98b784f9689fc6f07b0d98dab P 43d18c2b045b42029df4e57fd7704436: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:55:11.010355 14946 tablet_copy_service.cc:342] P bb14db373a304a2bbbeca9f1b6759d1e: Request end of tablet copy session 43d18c2b045b42029df4e57fd7704436-86a26ae98b784f9689fc6f07b0d98dab received from {username='slave'} at 127.31.161.195:54335
I20240502 01:55:11.010473 14946 tablet_copy_service.cc:433] P bb14db373a304a2bbbeca9f1b6759d1e: ending tablet copy session 43d18c2b045b42029df4e57fd7704436-86a26ae98b784f9689fc6f07b0d98dab on tablet 86a26ae98b784f9689fc6f07b0d98dab with peer 43d18c2b045b42029df4e57fd7704436
I20240502 01:55:11.012296 15705 tablet_copy_client.cc:783] T efc65c4da5f34c259006ce1834edac6c P c333763da41a4fd8b7b1397b88bff647: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:11.012626 15705 tablet_copy_client.cc:647] T efc65c4da5f34c259006ce1834edac6c P c333763da41a4fd8b7b1397b88bff647: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:11.014808 15705 tablet_copy_client.cc:523] T efc65c4da5f34c259006ce1834edac6c P c333763da41a4fd8b7b1397b88bff647: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:11.017796 15705 tablet_bootstrap.cc:492] T efc65c4da5f34c259006ce1834edac6c P c333763da41a4fd8b7b1397b88bff647: Bootstrap starting.
I20240502 01:55:11.018222 15698 ts_tablet_manager.cc:902] T 6d42de8e18d44481b051216dea2a6a55 P 43d18c2b045b42029df4e57fd7704436: Initiating tablet copy from peer bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196:34533)
I20240502 01:55:11.019317 15698 tablet_copy_client.cc:308] T 6d42de8e18d44481b051216dea2a6a55 P 43d18c2b045b42029df4e57fd7704436: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:34533
I20240502 01:55:11.019953 14946 tablet_copy_service.cc:140] P bb14db373a304a2bbbeca9f1b6759d1e: Received BeginTabletCopySession request for tablet 6d42de8e18d44481b051216dea2a6a55 from peer 43d18c2b045b42029df4e57fd7704436 ({username='slave'} at 127.31.161.195:54335)
I20240502 01:55:11.020097 14946 tablet_copy_service.cc:161] P bb14db373a304a2bbbeca9f1b6759d1e: Beginning new tablet copy session on tablet 6d42de8e18d44481b051216dea2a6a55 from peer 43d18c2b045b42029df4e57fd7704436 at {username='slave'} at 127.31.161.195:54335: session id = 43d18c2b045b42029df4e57fd7704436-6d42de8e18d44481b051216dea2a6a55
I20240502 01:55:11.022162 14946 tablet_copy_source_session.cc:205] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:11.023298 15698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d42de8e18d44481b051216dea2a6a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:11.029192 15698 tablet_copy_client.cc:783] T 6d42de8e18d44481b051216dea2a6a55 P 43d18c2b045b42029df4e57fd7704436: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:11.029198 15705 log.cc:824] T efc65c4da5f34c259006ce1834edac6c P c333763da41a4fd8b7b1397b88bff647: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:11.029430 15698 tablet_copy_client.cc:647] T 6d42de8e18d44481b051216dea2a6a55 P 43d18c2b045b42029df4e57fd7704436: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:11.030931 15698 tablet_copy_client.cc:523] T 6d42de8e18d44481b051216dea2a6a55 P 43d18c2b045b42029df4e57fd7704436: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:11.034111 15698 tablet_bootstrap.cc:492] T 6d42de8e18d44481b051216dea2a6a55 P 43d18c2b045b42029df4e57fd7704436: Bootstrap starting.
I20240502 01:55:11.044725 15717 ts_tablet_manager.cc:902] T 6654e083c8e94913aa2a8bd0ccb1af5a P c333763da41a4fd8b7b1397b88bff647: Initiating tablet copy from peer 382625cd16224c8aa404242500ab3bfa (127.31.161.193:45719)
I20240502 01:55:11.052590 15717 tablet_copy_client.cc:308] T 6654e083c8e94913aa2a8bd0ccb1af5a P c333763da41a4fd8b7b1397b88bff647: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.193:45719
I20240502 01:55:11.053786 14549 tablet_copy_service.cc:140] P 382625cd16224c8aa404242500ab3bfa: Received BeginTabletCopySession request for tablet 6654e083c8e94913aa2a8bd0ccb1af5a from peer c333763da41a4fd8b7b1397b88bff647 ({username='slave'} at 127.31.161.194:45553)
I20240502 01:55:11.053995 14549 tablet_copy_service.cc:161] P 382625cd16224c8aa404242500ab3bfa: Beginning new tablet copy session on tablet 6654e083c8e94913aa2a8bd0ccb1af5a from peer c333763da41a4fd8b7b1397b88bff647 at {username='slave'} at 127.31.161.194:45553: session id = c333763da41a4fd8b7b1397b88bff647-6654e083c8e94913aa2a8bd0ccb1af5a
I20240502 01:55:11.056085 14549 tablet_copy_source_session.cc:205] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:11.057535 15717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6654e083c8e94913aa2a8bd0ccb1af5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:11.064554 15717 tablet_copy_client.cc:783] T 6654e083c8e94913aa2a8bd0ccb1af5a P c333763da41a4fd8b7b1397b88bff647: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:11.064877 15717 tablet_copy_client.cc:647] T 6654e083c8e94913aa2a8bd0ccb1af5a P c333763da41a4fd8b7b1397b88bff647: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:11.069160 15717 tablet_copy_client.cc:523] T 6654e083c8e94913aa2a8bd0ccb1af5a P c333763da41a4fd8b7b1397b88bff647: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:11.069437 15724 ts_tablet_manager.cc:902] T beb3ef1600054db9903893cd89bd464c P c333763da41a4fd8b7b1397b88bff647: Initiating tablet copy from peer 382625cd16224c8aa404242500ab3bfa (127.31.161.193:45719)
I20240502 01:55:11.072973 15717 tablet_bootstrap.cc:492] T 6654e083c8e94913aa2a8bd0ccb1af5a P c333763da41a4fd8b7b1397b88bff647: Bootstrap starting.
I20240502 01:55:11.080829 15705 tablet_bootstrap.cc:492] T efc65c4da5f34c259006ce1834edac6c P c333763da41a4fd8b7b1397b88bff647: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:11.081439 15705 tablet_bootstrap.cc:492] T efc65c4da5f34c259006ce1834edac6c P c333763da41a4fd8b7b1397b88bff647: Bootstrap complete.
I20240502 01:55:11.082007 15705 ts_tablet_manager.cc:1367] T efc65c4da5f34c259006ce1834edac6c P c333763da41a4fd8b7b1397b88bff647: Time spent bootstrapping tablet: real 0.064s user 0.055s sys 0.005s
I20240502 01:55:11.082975 15698 tablet_bootstrap.cc:492] T 6d42de8e18d44481b051216dea2a6a55 P 43d18c2b045b42029df4e57fd7704436: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:11.083423 15698 tablet_bootstrap.cc:492] T 6d42de8e18d44481b051216dea2a6a55 P 43d18c2b045b42029df4e57fd7704436: Bootstrap complete.
I20240502 01:55:11.083675 15698 ts_tablet_manager.cc:1367] T 6d42de8e18d44481b051216dea2a6a55 P 43d18c2b045b42029df4e57fd7704436: Time spent bootstrapping tablet: real 0.050s user 0.044s sys 0.004s
I20240502 01:55:11.084504 15698 raft_consensus.cc:348] T 6d42de8e18d44481b051216dea2a6a55 P 43d18c2b045b42029df4e57fd7704436 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: true } }
I20240502 01:55:11.084749 15698 raft_consensus.cc:729] T 6d42de8e18d44481b051216dea2a6a55 P 43d18c2b045b42029df4e57fd7704436 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 43d18c2b045b42029df4e57fd7704436, State: Initialized, Role: LEARNER
I20240502 01:55:11.084964 15698 consensus_queue.cc:260] T 6d42de8e18d44481b051216dea2a6a55 P 43d18c2b045b42029df4e57fd7704436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: NON_VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: true } }
I20240502 01:55:11.086223 15698 ts_tablet_manager.cc:1398] T 6d42de8e18d44481b051216dea2a6a55 P 43d18c2b045b42029df4e57fd7704436: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:55:11.086943 14946 tablet_copy_service.cc:342] P bb14db373a304a2bbbeca9f1b6759d1e: Request end of tablet copy session 43d18c2b045b42029df4e57fd7704436-6d42de8e18d44481b051216dea2a6a55 received from {username='slave'} at 127.31.161.195:54335
I20240502 01:55:11.087117 14946 tablet_copy_service.cc:433] P bb14db373a304a2bbbeca9f1b6759d1e: ending tablet copy session 43d18c2b045b42029df4e57fd7704436-6d42de8e18d44481b051216dea2a6a55 on tablet 6d42de8e18d44481b051216dea2a6a55 with peer 43d18c2b045b42029df4e57fd7704436
I20240502 01:55:11.089362 15705 raft_consensus.cc:348] T efc65c4da5f34c259006ce1834edac6c P c333763da41a4fd8b7b1397b88bff647 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: true } }
I20240502 01:55:11.089911 15705 raft_consensus.cc:729] T efc65c4da5f34c259006ce1834edac6c P c333763da41a4fd8b7b1397b88bff647 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c333763da41a4fd8b7b1397b88bff647, State: Initialized, Role: LEARNER
I20240502 01:55:11.090600 15705 consensus_queue.cc:260] T efc65c4da5f34c259006ce1834edac6c P c333763da41a4fd8b7b1397b88bff647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: true } }
I20240502 01:55:11.095731 15705 ts_tablet_manager.cc:1398] T efc65c4da5f34c259006ce1834edac6c P c333763da41a4fd8b7b1397b88bff647: Time spent starting tablet: real 0.014s user 0.007s sys 0.004s
I20240502 01:55:11.096304 15724 tablet_copy_client.cc:308] T beb3ef1600054db9903893cd89bd464c P c333763da41a4fd8b7b1397b88bff647: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.193:45719
I20240502 01:55:11.097141 14549 tablet_copy_service.cc:342] P 382625cd16224c8aa404242500ab3bfa: Request end of tablet copy session c333763da41a4fd8b7b1397b88bff647-efc65c4da5f34c259006ce1834edac6c received from {username='slave'} at 127.31.161.194:45553
I20240502 01:55:11.097237 14548 tablet_copy_service.cc:140] P 382625cd16224c8aa404242500ab3bfa: Received BeginTabletCopySession request for tablet beb3ef1600054db9903893cd89bd464c from peer c333763da41a4fd8b7b1397b88bff647 ({username='slave'} at 127.31.161.194:45553)
I20240502 01:55:11.097332 14549 tablet_copy_service.cc:433] P 382625cd16224c8aa404242500ab3bfa: ending tablet copy session c333763da41a4fd8b7b1397b88bff647-efc65c4da5f34c259006ce1834edac6c on tablet efc65c4da5f34c259006ce1834edac6c with peer c333763da41a4fd8b7b1397b88bff647
I20240502 01:55:11.097934 14548 tablet_copy_service.cc:161] P 382625cd16224c8aa404242500ab3bfa: Beginning new tablet copy session on tablet beb3ef1600054db9903893cd89bd464c from peer c333763da41a4fd8b7b1397b88bff647 at {username='slave'} at 127.31.161.194:45553: session id = c333763da41a4fd8b7b1397b88bff647-beb3ef1600054db9903893cd89bd464c
I20240502 01:55:11.101269 14548 tablet_copy_source_session.cc:205] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:11.102557 15724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet beb3ef1600054db9903893cd89bd464c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:11.109915 15724 tablet_copy_client.cc:783] T beb3ef1600054db9903893cd89bd464c P c333763da41a4fd8b7b1397b88bff647: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:11.110217 15724 tablet_copy_client.cc:647] T beb3ef1600054db9903893cd89bd464c P c333763da41a4fd8b7b1397b88bff647: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:11.112067 15724 tablet_copy_client.cc:523] T beb3ef1600054db9903893cd89bd464c P c333763da41a4fd8b7b1397b88bff647: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:11.115460 15724 tablet_bootstrap.cc:492] T beb3ef1600054db9903893cd89bd464c P c333763da41a4fd8b7b1397b88bff647: Bootstrap starting.
I20240502 01:55:11.132064 15717 tablet_bootstrap.cc:492] T 6654e083c8e94913aa2a8bd0ccb1af5a P c333763da41a4fd8b7b1397b88bff647: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:11.132530 15717 tablet_bootstrap.cc:492] T 6654e083c8e94913aa2a8bd0ccb1af5a P c333763da41a4fd8b7b1397b88bff647: Bootstrap complete.
I20240502 01:55:11.132812 15717 ts_tablet_manager.cc:1367] T 6654e083c8e94913aa2a8bd0ccb1af5a P c333763da41a4fd8b7b1397b88bff647: Time spent bootstrapping tablet: real 0.060s user 0.047s sys 0.010s
I20240502 01:55:11.133869 15717 raft_consensus.cc:348] T 6654e083c8e94913aa2a8bd0ccb1af5a P c333763da41a4fd8b7b1397b88bff647 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: true } }
I20240502 01:55:11.134151 15717 raft_consensus.cc:729] T 6654e083c8e94913aa2a8bd0ccb1af5a P c333763da41a4fd8b7b1397b88bff647 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c333763da41a4fd8b7b1397b88bff647, State: Initialized, Role: LEARNER
I20240502 01:55:11.134384 15717 consensus_queue.cc:260] T 6654e083c8e94913aa2a8bd0ccb1af5a P c333763da41a4fd8b7b1397b88bff647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: true } }
I20240502 01:55:11.135551 15717 ts_tablet_manager.cc:1398] T 6654e083c8e94913aa2a8bd0ccb1af5a P c333763da41a4fd8b7b1397b88bff647: Time spent starting tablet: real 0.002s user 0.006s sys 0.000s
I20240502 01:55:11.136289 14548 tablet_copy_service.cc:342] P 382625cd16224c8aa404242500ab3bfa: Request end of tablet copy session c333763da41a4fd8b7b1397b88bff647-6654e083c8e94913aa2a8bd0ccb1af5a received from {username='slave'} at 127.31.161.194:45553
I20240502 01:55:11.136406 14548 tablet_copy_service.cc:433] P 382625cd16224c8aa404242500ab3bfa: ending tablet copy session c333763da41a4fd8b7b1397b88bff647-6654e083c8e94913aa2a8bd0ccb1af5a on tablet 6654e083c8e94913aa2a8bd0ccb1af5a with peer c333763da41a4fd8b7b1397b88bff647
I20240502 01:55:11.157423 15724 tablet_bootstrap.cc:492] T beb3ef1600054db9903893cd89bd464c P c333763da41a4fd8b7b1397b88bff647: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:11.157791 15724 tablet_bootstrap.cc:492] T beb3ef1600054db9903893cd89bd464c P c333763da41a4fd8b7b1397b88bff647: Bootstrap complete.
I20240502 01:55:11.158048 15724 ts_tablet_manager.cc:1367] T beb3ef1600054db9903893cd89bd464c P c333763da41a4fd8b7b1397b88bff647: Time spent bootstrapping tablet: real 0.043s user 0.036s sys 0.007s
I20240502 01:55:11.158707 15724 raft_consensus.cc:348] T beb3ef1600054db9903893cd89bd464c P c333763da41a4fd8b7b1397b88bff647 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: true } }
I20240502 01:55:11.158910 15724 raft_consensus.cc:729] T beb3ef1600054db9903893cd89bd464c P c333763da41a4fd8b7b1397b88bff647 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c333763da41a4fd8b7b1397b88bff647, State: Initialized, Role: LEARNER
I20240502 01:55:11.159094 15724 consensus_queue.cc:260] T beb3ef1600054db9903893cd89bd464c P c333763da41a4fd8b7b1397b88bff647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: NON_VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: true } }
I20240502 01:55:11.159592 15724 ts_tablet_manager.cc:1398] T beb3ef1600054db9903893cd89bd464c P c333763da41a4fd8b7b1397b88bff647: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:55:11.160395 14548 tablet_copy_service.cc:342] P 382625cd16224c8aa404242500ab3bfa: Request end of tablet copy session c333763da41a4fd8b7b1397b88bff647-beb3ef1600054db9903893cd89bd464c received from {username='slave'} at 127.31.161.194:45553
I20240502 01:55:11.160553 14548 tablet_copy_service.cc:433] P 382625cd16224c8aa404242500ab3bfa: ending tablet copy session c333763da41a4fd8b7b1397b88bff647-beb3ef1600054db9903893cd89bd464c on tablet beb3ef1600054db9903893cd89bd464c with peer c333763da41a4fd8b7b1397b88bff647
I20240502 01:55:11.395339 15399 raft_consensus.cc:1203] T 31e69931401143bbb018e06772c15fd8 P 43d18c2b045b42029df4e57fd7704436 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:55:11.445588 15267 raft_consensus.cc:1203] T efc65c4da5f34c259006ce1834edac6c P c333763da41a4fd8b7b1397b88bff647 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:55:11.471081 15399 raft_consensus.cc:1203] T 86a26ae98b784f9689fc6f07b0d98dab P 43d18c2b045b42029df4e57fd7704436 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:55:11.486883 15267 raft_consensus.cc:1203] T beb3ef1600054db9903893cd89bd464c P c333763da41a4fd8b7b1397b88bff647 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:55:11.592432 15267 raft_consensus.cc:1203] T 6654e083c8e94913aa2a8bd0ccb1af5a P c333763da41a4fd8b7b1397b88bff647 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:55:11.630756 15399 raft_consensus.cc:1203] T 6d42de8e18d44481b051216dea2a6a55 P 43d18c2b045b42029df4e57fd7704436 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 01:55:11.786839 15114 raft_consensus.cc:1050] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e: attempting to promote NON_VOTER 43d18c2b045b42029df4e57fd7704436 to VOTER
I20240502 01:55:11.787663 15114 consensus_queue.cc:237] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } }
I20240502 01:55:11.789739 14528 raft_consensus.cc:1261] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Refusing update from remote peer bb14db373a304a2bbbeca9f1b6759d1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:11.789791 15399 raft_consensus.cc:1261] T 31e69931401143bbb018e06772c15fd8 P 43d18c2b045b42029df4e57fd7704436 [term 1 LEARNER]: Refusing update from remote peer bb14db373a304a2bbbeca9f1b6759d1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:11.790323 15623 consensus_queue.cc:1035] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:11.790664 15779 consensus_queue.cc:1035] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:11.793201 15623 raft_consensus.cc:2932] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 43d18c2b045b42029df4e57fd7704436 (127.31.161.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } } }
I20240502 01:55:11.793948 14528 raft_consensus.cc:2932] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 43d18c2b045b42029df4e57fd7704436 (127.31.161.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } } }
I20240502 01:55:11.794975 15399 raft_consensus.cc:2932] T 31e69931401143bbb018e06772c15fd8 P 43d18c2b045b42029df4e57fd7704436 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 43d18c2b045b42029df4e57fd7704436 (127.31.161.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } } }
I20240502 01:55:11.799423 14379 catalog_manager.cc:5484] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa reported cstate change: config changed from index 4 to 5, 43d18c2b045b42029df4e57fd7704436 (127.31.161.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } } }
I20240502 01:55:11.806071 14925 consensus_queue.cc:237] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } }
I20240502 01:55:11.807632 15399 raft_consensus.cc:1261] T 31e69931401143bbb018e06772c15fd8 P 43d18c2b045b42029df4e57fd7704436 [term 1 FOLLOWER]: Refusing update from remote peer bb14db373a304a2bbbeca9f1b6759d1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:11.808107 15623 consensus_queue.cc:1035] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:11.810050 15779 raft_consensus.cc:2932] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 382625cd16224c8aa404242500ab3bfa (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } } }
I20240502 01:55:11.810627 15399 raft_consensus.cc:2932] T 31e69931401143bbb018e06772c15fd8 P 43d18c2b045b42029df4e57fd7704436 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 382625cd16224c8aa404242500ab3bfa (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } } }
I20240502 01:55:11.812832 14369 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 31e69931401143bbb018e06772c15fd8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:11.814177 14384 catalog_manager.cc:5484] T 31e69931401143bbb018e06772c15fd8 P bb14db373a304a2bbbeca9f1b6759d1e reported cstate change: config changed from index 5 to 6, VOTER 382625cd16224c8aa404242500ab3bfa (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:11.817870 14508 tablet_service.cc:1514] Processing DeleteTablet for tablet 31e69931401143bbb018e06772c15fd8 with delete_type TABLET_DATA_TOMBSTONED (TS 382625cd16224c8aa404242500ab3bfa not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49622
I20240502 01:55:11.818841 15831 tablet_replica.cc:330] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa: stopping tablet replica
I20240502 01:55:11.819257 15831 raft_consensus.cc:2229] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:11.819624 15831 raft_consensus.cc:2258] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:11.821003 15831 ts_tablet_manager.cc:1875] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:11.825593 15831 ts_tablet_manager.cc:1888] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:55:11.825708 15831 log.cc:1196] T 31e69931401143bbb018e06772c15fd8 P 382625cd16224c8aa404242500ab3bfa: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/31e69931401143bbb018e06772c15fd8
I20240502 01:55:11.826517 14370 catalog_manager.cc:4830] TS 382625cd16224c8aa404242500ab3bfa (127.31.161.193:45719): tablet 31e69931401143bbb018e06772c15fd8 (table rebalance_test_table_1 [id=ab8163db47cd4b329cd331b45db9d06c]) successfully deleted
I20240502 01:55:11.893397 15793 raft_consensus.cc:1050] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa: attempting to promote NON_VOTER c333763da41a4fd8b7b1397b88bff647 to VOTER
I20240502 01:55:11.894570 15793 consensus_queue.cc:237] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } }
I20240502 01:55:11.897316 15267 raft_consensus.cc:1261] T beb3ef1600054db9903893cd89bd464c P c333763da41a4fd8b7b1397b88bff647 [term 1 LEARNER]: Refusing update from remote peer 382625cd16224c8aa404242500ab3bfa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:11.897972 15793 consensus_queue.cc:1035] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:11.898151 14925 raft_consensus.cc:1261] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Refusing update from remote peer 382625cd16224c8aa404242500ab3bfa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:11.898846 15793 consensus_queue.cc:1035] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:11.902740 15792 raft_consensus.cc:2932] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c333763da41a4fd8b7b1397b88bff647 (127.31.161.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } } }
I20240502 01:55:11.905570 15267 raft_consensus.cc:2932] T beb3ef1600054db9903893cd89bd464c P c333763da41a4fd8b7b1397b88bff647 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c333763da41a4fd8b7b1397b88bff647 (127.31.161.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } } }
I20240502 01:55:11.909159 14384 catalog_manager.cc:5484] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa reported cstate change: config changed from index 4 to 5, c333763da41a4fd8b7b1397b88bff647 (127.31.161.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "382625cd16224c8aa404242500ab3bfa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:11.911041 14925 raft_consensus.cc:2932] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c333763da41a4fd8b7b1397b88bff647 (127.31.161.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } } }
I20240502 01:55:11.914256 14528 consensus_queue.cc:237] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } }
I20240502 01:55:11.916358 15267 raft_consensus.cc:1261] T beb3ef1600054db9903893cd89bd464c P c333763da41a4fd8b7b1397b88bff647 [term 1 FOLLOWER]: Refusing update from remote peer 382625cd16224c8aa404242500ab3bfa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:11.916918 15792 consensus_queue.cc:1035] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:11.919236 15599 raft_consensus.cc:2932] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } } }
I20240502 01:55:11.919909 15267 raft_consensus.cc:2932] T beb3ef1600054db9903893cd89bd464c P c333763da41a4fd8b7b1397b88bff647 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } } }
I20240502 01:55:11.922850 14370 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet beb3ef1600054db9903893cd89bd464c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:11.924743 14383 catalog_manager.cc:5484] T beb3ef1600054db9903893cd89bd464c P 382625cd16224c8aa404242500ab3bfa reported cstate change: config changed from index 5 to 6, VOTER bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "382625cd16224c8aa404242500ab3bfa" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:11.928709 14905 tablet_service.cc:1514] Processing DeleteTablet for tablet beb3ef1600054db9903893cd89bd464c with delete_type TABLET_DATA_TOMBSTONED (TS bb14db373a304a2bbbeca9f1b6759d1e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59056
I20240502 01:55:11.930619 15851 tablet_replica.cc:330] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e: stopping tablet replica
I20240502 01:55:11.931103 15851 raft_consensus.cc:2229] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:11.931581 15851 raft_consensus.cc:2258] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:11.933156 15851 ts_tablet_manager.cc:1875] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:11.938571 15851 ts_tablet_manager.cc:1888] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:55:11.938728 15851 log.cc:1196] T beb3ef1600054db9903893cd89bd464c P bb14db373a304a2bbbeca9f1b6759d1e: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/beb3ef1600054db9903893cd89bd464c
I20240502 01:55:11.939546 14369 catalog_manager.cc:4830] TS bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196:34533): tablet beb3ef1600054db9903893cd89bd464c (table rebalance_test_table_1 [id=ab8163db47cd4b329cd331b45db9d06c]) successfully deleted
I20240502 01:55:11.943092 15792 raft_consensus.cc:1050] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa: attempting to promote NON_VOTER c333763da41a4fd8b7b1397b88bff647 to VOTER
I20240502 01:55:11.943920 15792 consensus_queue.cc:237] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } }
I20240502 01:55:11.946009 15267 raft_consensus.cc:1261] T efc65c4da5f34c259006ce1834edac6c P c333763da41a4fd8b7b1397b88bff647 [term 1 LEARNER]: Refusing update from remote peer 382625cd16224c8aa404242500ab3bfa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:11.946439 14925 raft_consensus.cc:1261] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Refusing update from remote peer 382625cd16224c8aa404242500ab3bfa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:11.946645 15793 consensus_queue.cc:1035] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:11.946995 15792 consensus_queue.cc:1035] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:11.950371 15793 raft_consensus.cc:2932] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c333763da41a4fd8b7b1397b88bff647 (127.31.161.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } } }
I20240502 01:55:11.951220 14925 raft_consensus.cc:2932] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c333763da41a4fd8b7b1397b88bff647 (127.31.161.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } } }
I20240502 01:55:11.954099 15267 raft_consensus.cc:2932] T efc65c4da5f34c259006ce1834edac6c P c333763da41a4fd8b7b1397b88bff647 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c333763da41a4fd8b7b1397b88bff647 (127.31.161.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } } }
I20240502 01:55:11.958806 14384 catalog_manager.cc:5484] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e reported cstate change: config changed from index 4 to 5, c333763da41a4fd8b7b1397b88bff647 (127.31.161.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "382625cd16224c8aa404242500ab3bfa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } } }
I20240502 01:55:11.963814 14529 consensus_queue.cc:237] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } }
I20240502 01:55:11.966650 15267 raft_consensus.cc:1261] T efc65c4da5f34c259006ce1834edac6c P c333763da41a4fd8b7b1397b88bff647 [term 1 FOLLOWER]: Refusing update from remote peer 382625cd16224c8aa404242500ab3bfa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:11.967238 15793 consensus_queue.cc:1035] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:11.969461 15792 raft_consensus.cc:2932] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } } }
I20240502 01:55:11.970158 15267 raft_consensus.cc:2932] T efc65c4da5f34c259006ce1834edac6c P c333763da41a4fd8b7b1397b88bff647 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } } }
I20240502 01:55:11.973224 14370 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet efc65c4da5f34c259006ce1834edac6c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:11.974607 14383 catalog_manager.cc:5484] T efc65c4da5f34c259006ce1834edac6c P 382625cd16224c8aa404242500ab3bfa reported cstate change: config changed from index 5 to 6, VOTER bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "382625cd16224c8aa404242500ab3bfa" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:11.978562 14905 tablet_service.cc:1514] Processing DeleteTablet for tablet efc65c4da5f34c259006ce1834edac6c with delete_type TABLET_DATA_TOMBSTONED (TS bb14db373a304a2bbbeca9f1b6759d1e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59056
I20240502 01:55:11.978938 15851 tablet_replica.cc:330] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e: stopping tablet replica
I20240502 01:55:11.979147 15851 raft_consensus.cc:2229] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:11.979409 15851 raft_consensus.cc:2258] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:11.980506 15851 ts_tablet_manager.cc:1875] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:11.985271 15851 ts_tablet_manager.cc:1888] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:55:11.985395 15851 log.cc:1196] T efc65c4da5f34c259006ce1834edac6c P bb14db373a304a2bbbeca9f1b6759d1e: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/efc65c4da5f34c259006ce1834edac6c
I20240502 01:55:11.986140 14369 catalog_manager.cc:4830] TS bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196:34533): tablet efc65c4da5f34c259006ce1834edac6c (table rebalance_test_table_2 [id=1ca71c9931094ef5af1861989e823dcb]) successfully deleted
I20240502 01:55:12.035867 15623 raft_consensus.cc:1050] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e: attempting to promote NON_VOTER 43d18c2b045b42029df4e57fd7704436 to VOTER
I20240502 01:55:12.036492 15623 consensus_queue.cc:237] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } }
I20240502 01:55:12.038444 14529 raft_consensus.cc:1261] T 86a26ae98b784f9689fc6f07b0d98dab P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Refusing update from remote peer bb14db373a304a2bbbeca9f1b6759d1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:12.038427 15399 raft_consensus.cc:1261] T 86a26ae98b784f9689fc6f07b0d98dab P 43d18c2b045b42029df4e57fd7704436 [term 1 LEARNER]: Refusing update from remote peer bb14db373a304a2bbbeca9f1b6759d1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:12.039055 15779 consensus_queue.cc:1035] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:12.039351 15114 consensus_queue.cc:1035] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:12.042057 15623 raft_consensus.cc:2932] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 43d18c2b045b42029df4e57fd7704436 (127.31.161.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } } }
I20240502 01:55:12.042685 15399 raft_consensus.cc:2932] T 86a26ae98b784f9689fc6f07b0d98dab P 43d18c2b045b42029df4e57fd7704436 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 43d18c2b045b42029df4e57fd7704436 (127.31.161.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } } }
I20240502 01:55:12.046052 14529 raft_consensus.cc:2932] T 86a26ae98b784f9689fc6f07b0d98dab P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 43d18c2b045b42029df4e57fd7704436 (127.31.161.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } } }
I20240502 01:55:12.047355 14379 catalog_manager.cc:5484] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e reported cstate change: config changed from index 4 to 5, 43d18c2b045b42029df4e57fd7704436 (127.31.161.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:12.051477 14925 consensus_queue.cc:237] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } }
I20240502 01:55:12.052861 15399 raft_consensus.cc:1261] T 86a26ae98b784f9689fc6f07b0d98dab P 43d18c2b045b42029df4e57fd7704436 [term 1 FOLLOWER]: Refusing update from remote peer bb14db373a304a2bbbeca9f1b6759d1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:12.053407 15623 consensus_queue.cc:1035] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:12.055457 15779 raft_consensus.cc:2932] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 382625cd16224c8aa404242500ab3bfa (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } } }
I20240502 01:55:12.056093 15399 raft_consensus.cc:2932] T 86a26ae98b784f9689fc6f07b0d98dab P 43d18c2b045b42029df4e57fd7704436 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 382625cd16224c8aa404242500ab3bfa (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } } }
I20240502 01:55:12.058930 14369 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 86a26ae98b784f9689fc6f07b0d98dab with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:12.060580 14384 catalog_manager.cc:5484] T 86a26ae98b784f9689fc6f07b0d98dab P bb14db373a304a2bbbeca9f1b6759d1e reported cstate change: config changed from index 5 to 6, VOTER 382625cd16224c8aa404242500ab3bfa (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:12.063833 14508 tablet_service.cc:1514] Processing DeleteTablet for tablet 86a26ae98b784f9689fc6f07b0d98dab with delete_type TABLET_DATA_TOMBSTONED (TS 382625cd16224c8aa404242500ab3bfa not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49622
I20240502 01:55:12.064242 15831 tablet_replica.cc:330] T 86a26ae98b784f9689fc6f07b0d98dab P 382625cd16224c8aa404242500ab3bfa: stopping tablet replica
I20240502 01:55:12.064483 15831 raft_consensus.cc:2229] T 86a26ae98b784f9689fc6f07b0d98dab P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:12.064803 15831 raft_consensus.cc:2258] T 86a26ae98b784f9689fc6f07b0d98dab P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:12.065671 15831 ts_tablet_manager.cc:1875] T 86a26ae98b784f9689fc6f07b0d98dab P 382625cd16224c8aa404242500ab3bfa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:12.070225 15831 ts_tablet_manager.cc:1888] T 86a26ae98b784f9689fc6f07b0d98dab P 382625cd16224c8aa404242500ab3bfa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:55:12.070343 15831 log.cc:1196] T 86a26ae98b784f9689fc6f07b0d98dab P 382625cd16224c8aa404242500ab3bfa: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/86a26ae98b784f9689fc6f07b0d98dab
I20240502 01:55:12.071069 14370 catalog_manager.cc:4830] TS 382625cd16224c8aa404242500ab3bfa (127.31.161.193:45719): tablet 86a26ae98b784f9689fc6f07b0d98dab (table rebalance_test_table_2 [id=1ca71c9931094ef5af1861989e823dcb]) successfully deleted
I20240502 01:55:12.210582 15792 raft_consensus.cc:1050] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa: attempting to promote NON_VOTER c333763da41a4fd8b7b1397b88bff647 to VOTER
I20240502 01:55:12.211405 15792 consensus_queue.cc:237] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } }
I20240502 01:55:12.213654 14925 raft_consensus.cc:1261] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Refusing update from remote peer 382625cd16224c8aa404242500ab3bfa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:12.213660 15267 raft_consensus.cc:1261] T 6654e083c8e94913aa2a8bd0ccb1af5a P c333763da41a4fd8b7b1397b88bff647 [term 1 LEARNER]: Refusing update from remote peer 382625cd16224c8aa404242500ab3bfa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:12.214249 15793 consensus_queue.cc:1035] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:12.214607 15792 consensus_queue.cc:1035] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:12.217336 15792 raft_consensus.cc:2932] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c333763da41a4fd8b7b1397b88bff647 (127.31.161.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } } }
I20240502 01:55:12.218148 14925 raft_consensus.cc:2932] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c333763da41a4fd8b7b1397b88bff647 (127.31.161.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } } }
I20240502 01:55:12.219086 15267 raft_consensus.cc:2932] T 6654e083c8e94913aa2a8bd0ccb1af5a P c333763da41a4fd8b7b1397b88bff647 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c333763da41a4fd8b7b1397b88bff647 (127.31.161.194) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } } }
I20240502 01:55:12.222110 14383 catalog_manager.cc:5484] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa reported cstate change: config changed from index 4 to 5, c333763da41a4fd8b7b1397b88bff647 (127.31.161.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "382625cd16224c8aa404242500ab3bfa" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:12.226446 14529 consensus_queue.cc:237] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } }
I20240502 01:55:12.227782 15267 raft_consensus.cc:1261] T 6654e083c8e94913aa2a8bd0ccb1af5a P c333763da41a4fd8b7b1397b88bff647 [term 1 FOLLOWER]: Refusing update from remote peer 382625cd16224c8aa404242500ab3bfa: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:12.228216 15599 consensus_queue.cc:1035] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:12.230376 15792 raft_consensus.cc:2932] T 6654e083c8e94913aa2a8bd0ccb1af5a P 382625cd16224c8aa404242500ab3bfa [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } } }
I20240502 01:55:12.230912 15267 raft_consensus.cc:2932] T 6654e083c8e94913aa2a8bd0ccb1af5a P c333763da41a4fd8b7b1397b88bff647 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } } }
I20240502 01:55:12.234401 14370 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6654e083c8e94913aa2a8bd0ccb1af5a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:12.235074 14379 catalog_manager.cc:5484] T 6654e083c8e94913aa2a8bd0ccb1af5a P c333763da41a4fd8b7b1397b88bff647 reported cstate change: config changed from index 5 to 6, VOTER bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "382625cd16224c8aa404242500ab3bfa" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } } peers { permanent_uuid: "c333763da41a4fd8b7b1397b88bff647" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 41573 } attrs { promote: false } } }
I20240502 01:55:12.239915 14905 tablet_service.cc:1514] Processing DeleteTablet for tablet 6654e083c8e94913aa2a8bd0ccb1af5a with delete_type TABLET_DATA_TOMBSTONED (TS bb14db373a304a2bbbeca9f1b6759d1e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:59056
I20240502 01:55:12.240288 15851 tablet_replica.cc:330] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e: stopping tablet replica
I20240502 01:55:12.240497 15851 raft_consensus.cc:2229] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:12.240878 15851 raft_consensus.cc:2258] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:12.241834 15851 ts_tablet_manager.cc:1875] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:12.246637 15851 ts_tablet_manager.cc:1888] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:55:12.246758 15623 raft_consensus.cc:1050] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e: attempting to promote NON_VOTER 43d18c2b045b42029df4e57fd7704436 to VOTER
I20240502 01:55:12.246791 15851 log.cc:1196] T 6654e083c8e94913aa2a8bd0ccb1af5a P bb14db373a304a2bbbeca9f1b6759d1e: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/6654e083c8e94913aa2a8bd0ccb1af5a
I20240502 01:55:12.247458 15623 consensus_queue.cc:237] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } }
I20240502 01:55:12.247586 14369 catalog_manager.cc:4830] TS bb14db373a304a2bbbeca9f1b6759d1e (127.31.161.196:34533): tablet 6654e083c8e94913aa2a8bd0ccb1af5a (table rebalance_test_table_0 [id=1ba208bc11da43d2a765c6489c0daa5d]) successfully deleted
I20240502 01:55:12.249644 14529 raft_consensus.cc:1261] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Refusing update from remote peer bb14db373a304a2bbbeca9f1b6759d1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:12.249868 15399 raft_consensus.cc:1261] T 6d42de8e18d44481b051216dea2a6a55 P 43d18c2b045b42029df4e57fd7704436 [term 1 LEARNER]: Refusing update from remote peer bb14db373a304a2bbbeca9f1b6759d1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:12.250216 15114 consensus_queue.cc:1035] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:12.251397 15114 consensus_queue.cc:1035] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:12.253430 15779 raft_consensus.cc:2932] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 43d18c2b045b42029df4e57fd7704436 (127.31.161.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } } }
I20240502 01:55:12.254127 14529 raft_consensus.cc:2932] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 43d18c2b045b42029df4e57fd7704436 (127.31.161.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } } }
I20240502 01:55:12.260491 14384 catalog_manager.cc:5484] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e reported cstate change: config changed from index 4 to 5, 43d18c2b045b42029df4e57fd7704436 (127.31.161.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:12.261467 15399 raft_consensus.cc:2932] T 6d42de8e18d44481b051216dea2a6a55 P 43d18c2b045b42029df4e57fd7704436 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 43d18c2b045b42029df4e57fd7704436 (127.31.161.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "382625cd16224c8aa404242500ab3bfa" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 45719 } attrs { replace: true } } peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } } }
I20240502 01:55:12.268596 14925 consensus_queue.cc:237] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } }
I20240502 01:55:12.270221 15399 raft_consensus.cc:1261] T 6d42de8e18d44481b051216dea2a6a55 P 43d18c2b045b42029df4e57fd7704436 [term 1 FOLLOWER]: Refusing update from remote peer bb14db373a304a2bbbeca9f1b6759d1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:12.270884 15114 consensus_queue.cc:1035] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:12.272863 15779 raft_consensus.cc:2932] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 382625cd16224c8aa404242500ab3bfa (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } } }
I20240502 01:55:12.273588 15399 raft_consensus.cc:2932] T 6d42de8e18d44481b051216dea2a6a55 P 43d18c2b045b42029df4e57fd7704436 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 382625cd16224c8aa404242500ab3bfa (127.31.161.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } } }
I20240502 01:55:12.276611 14369 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6d42de8e18d44481b051216dea2a6a55 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:12.278338 14379 catalog_manager.cc:5484] T 6d42de8e18d44481b051216dea2a6a55 P bb14db373a304a2bbbeca9f1b6759d1e reported cstate change: config changed from index 5 to 6, VOTER 382625cd16224c8aa404242500ab3bfa (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "bb14db373a304a2bbbeca9f1b6759d1e" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 34533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43d18c2b045b42029df4e57fd7704436" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 32901 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:12.282446 14508 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d42de8e18d44481b051216dea2a6a55 with delete_type TABLET_DATA_TOMBSTONED (TS 382625cd16224c8aa404242500ab3bfa not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49622
I20240502 01:55:12.282855 15831 tablet_replica.cc:330] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa: stopping tablet replica
I20240502 01:55:12.283048 15831 raft_consensus.cc:2229] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:12.283421 15831 raft_consensus.cc:2258] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:12.284485 15831 ts_tablet_manager.cc:1875] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:12.289886 15831 ts_tablet_manager.cc:1888] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 01:55:12.290046 15831 log.cc:1196] T 6d42de8e18d44481b051216dea2a6a55 P 382625cd16224c8aa404242500ab3bfa: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/6d42de8e18d44481b051216dea2a6a55
I20240502 01:55:12.290926 14370 catalog_manager.cc:4830] TS 382625cd16224c8aa404242500ab3bfa (127.31.161.193:45719): tablet 6d42de8e18d44481b051216dea2a6a55 (table rebalance_test_table_0 [id=1ba208bc11da43d2a765c6489c0daa5d]) successfully deleted
I20240502 01:55:12.389843 14508 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:12.403473 15379 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:12.404122 15247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:12.414933 14905 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:12.564358 14508 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:12.585976 14905 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:12.590516 15247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:12.602380 15379 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:12.734604 14508 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:12.751230 14905 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:12.753494 15379 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:12.759723 15247 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:12.925948 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 14446
I20240502 01:55:12.939214 14569 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:13.069155 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 14446
W20240502 01:55:13.090149 15199 connection.cc:535] server connection from 127.31.161.193:51599 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:55:13.091006 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 15184
I20240502 01:55:13.097293 15306 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:13.202489 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 15184
I20240502 01:55:13.220439 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 15316
I20240502 01:55:13.226357 15440 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 01:55:13.278774 14860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.161.193:45719: connect: Connection refused (error 111)
W20240502 01:55:13.282181 14860 consensus_peers.cc:485] T 273daa0829624e8bab379a0867748928 P bb14db373a304a2bbbeca9f1b6759d1e -> Peer 382625cd16224c8aa404242500ab3bfa (127.31.161.193:45719): Couldn't send request to peer 382625cd16224c8aa404242500ab3bfa. Status: Network error: Client connection negotiation failed: client connection to 127.31.161.193:45719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:13.333230 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 15316
W20240502 01:55:13.350584 14371 connection.cc:535] server connection from 127.31.161.195:59621 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 01:55:13.350831 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 14842
I20240502 01:55:13.356582 14966 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:13.480427 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 14842
I20240502 01:55:13.500018 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 14353
I20240502 01:55:13.501209 14414 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:13.634861 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 14353
2024-05-02T01:55:13Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.Rebalance/idx7_rf2_t3_l_1_2_1 (14581 ms)
[ RUN ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx6_rf7_t7_l_3_4_3
I20240502 01:55:13.683908 32391 test_util.cc:268] Using random seed: -455388773
I20240502 01:55:13.685858 32391 ts_itest-base.cc:112] Starting cluster with:
I20240502 01:55:13.685940 32391 ts_itest-base.cc:113] --------------
I20240502 01:55:13.685976 32391 ts_itest-base.cc:114] 10 tablet servers
I20240502 01:55:13.686024 32391 ts_itest-base.cc:115] 7 replicas per TS
I20240502 01:55:13.686060 32391 ts_itest-base.cc:116] --------------
2024-05-02T01:55:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T01:55:13Z Disabled control of system clock
I20240502 01:55:13.733168 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.161.254:46705
--webserver_interface=127.31.161.254
--webserver_port=0
--builtin_ntp_servers=127.31.161.212:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.161.254:46705
--location_mapping_cmd=/tmp/dist-test-taskMchxvY/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:4 --map /C:3
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240502 01:55:14.062064 16052 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 01:55:14.062417 16052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:14.062515 16052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:14.062613 16052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:14.070367 16052 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:14.070502 16052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 01:55:14.070600 16052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:14.070663 16052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 01:55:14.070748 16052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 01:55:14.079917 16052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskMchxvY/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:4 --map /C:3
--ipki_ca_key_size=768
--master_addresses=127.31.161.254:46705
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.161.254:46705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.31.161.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:14.081270 16052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:14.082964 16052 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:14.093269 16057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:14.093827 16058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:14.094250 16060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:14.094516 16052 server_base.cc:1034] running on GCE node
I20240502 01:55:14.095248 16052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:14.096670 16052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:14.097887 16052 hybrid_clock.cc:648] HybridClock initialized: now 1714614914097809 us; error 44 us; skew 500 ppm
I20240502 01:55:14.098191 16052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:14.100419 16052 webserver.cc:460] Webserver started at http://127.31.161.254:39171/ using document root <none> and password file <none>
I20240502 01:55:14.101166 16052 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:14.101295 16052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:14.101588 16052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:14.104143 16052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "290b6a9770a44575941778c84ccaad78"
format_stamp: "Formatted at 2024-05-02 01:55:14 on dist-test-slave-06l8"
I20240502 01:55:14.104904 16052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "290b6a9770a44575941778c84ccaad78"
format_stamp: "Formatted at 2024-05-02 01:55:14 on dist-test-slave-06l8"
I20240502 01:55:14.110059 16052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240502 01:55:14.113394 16066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:14.114751 16052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:55:14.114912 16052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
uuid: "290b6a9770a44575941778c84ccaad78"
format_stamp: "Formatted at 2024-05-02 01:55:14 on dist-test-slave-06l8"
I20240502 01:55:14.115083 16052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:14.129925 16052 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:55:14.130754 16052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:14.131024 16052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:14.148129 16052 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.254:46705
I20240502 01:55:14.148159 16117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.254:46705 every 8 connection(s)
I20240502 01:55:14.149499 16052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 01:55:14.150686 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 16052
I20240502 01:55:14.150940 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 01:55:14.154099 16118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:14.165257 16118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78: Bootstrap starting.
I20240502 01:55:14.168079 16118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:14.169049 16118 log.cc:824] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:14.171653 16118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78: No bootstrap required, opened a new log
I20240502 01:55:14.177810 16118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290b6a9770a44575941778c84ccaad78" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 46705 } }
I20240502 01:55:14.178138 16118 raft_consensus.cc:374] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:14.178231 16118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290b6a9770a44575941778c84ccaad78, State: Initialized, Role: FOLLOWER
I20240502 01:55:14.178800 16118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290b6a9770a44575941778c84ccaad78" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 46705 } }
I20240502 01:55:14.178997 16118 raft_consensus.cc:388] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 01:55:14.179116 16118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 01:55:14.179255 16118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:14.181340 16118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290b6a9770a44575941778c84ccaad78" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 46705 } }
I20240502 01:55:14.181913 16118 leader_election.cc:302] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 290b6a9770a44575941778c84ccaad78; no voters:
I20240502 01:55:14.182643 16118 leader_election.cc:288] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 01:55:14.182778 16123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:14.183420 16123 raft_consensus.cc:686] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78 [term 1 LEADER]: Becoming Leader. State: Replica: 290b6a9770a44575941778c84ccaad78, State: Running, Role: LEADER
I20240502 01:55:14.183984 16123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290b6a9770a44575941778c84ccaad78" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 46705 } }
I20240502 01:55:14.184705 16118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 01:55:14.186664 16124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "290b6a9770a44575941778c84ccaad78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290b6a9770a44575941778c84ccaad78" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 46705 } } }
I20240502 01:55:14.186772 16125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 290b6a9770a44575941778c84ccaad78. Latest consensus state: current_term: 1 leader_uuid: "290b6a9770a44575941778c84ccaad78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "290b6a9770a44575941778c84ccaad78" member_type: VOTER last_known_addr { host: "127.31.161.254" port: 46705 } } }
I20240502 01:55:14.187043 16124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78 [sys.catalog]: This master's current role is: LEADER
I20240502 01:55:14.187036 16125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78 [sys.catalog]: This master's current role is: LEADER
I20240502 01:55:14.189363 16130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 01:55:14.193967 16130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 01:55:14.203140 16130 catalog_manager.cc:1347] Generated new cluster ID: bec332b558034d95b1924cade7aed59a
I20240502 01:55:14.203263 16130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 01:55:14.217490 16130 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 01:55:14.219247 16130 catalog_manager.cc:1504] Loading token signing keys...
I20240502 01:55:14.237927 16130 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 290b6a9770a44575941778c84ccaad78: Generated new TSK 0
I20240502 01:55:14.239058 16130 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 01:55:14.244633 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.193:0
--local_ip_for_outbound_sockets=127.31.161.193
--webserver_interface=127.31.161.193
--webserver_port=0
--tserver_master_addrs=127.31.161.254:46705
--builtin_ntp_servers=127.31.161.212:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:55:14.580789 16142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:14.581115 16142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:14.581267 16142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:14.589234 16142 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:14.589385 16142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:14.589572 16142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.193
I20240502 01:55:14.598865 16142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.31.161.193
--webserver_port=0
--tserver_master_addrs=127.31.161.254:46705
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.193
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:14.600211 16142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:14.602000 16142 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:14.614002 16147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:14.614194 16148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:14.614853 16151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:14.615090 16142 server_base.cc:1034] running on GCE node
I20240502 01:55:14.615617 16142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:14.616528 16142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:14.617727 16142 hybrid_clock.cc:648] HybridClock initialized: now 1714614914617679 us; error 35 us; skew 500 ppm
I20240502 01:55:14.618017 16142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:14.620134 16142 webserver.cc:460] Webserver started at http://127.31.161.193:37985/ using document root <none> and password file <none>
I20240502 01:55:14.620808 16142 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:14.620900 16142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:14.621204 16142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:14.623648 16142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cdd4d50250184223992325c86cb6c68e"
format_stamp: "Formatted at 2024-05-02 01:55:14 on dist-test-slave-06l8"
I20240502 01:55:14.624347 16142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cdd4d50250184223992325c86cb6c68e"
format_stamp: "Formatted at 2024-05-02 01:55:14 on dist-test-slave-06l8"
I20240502 01:55:14.629251 16142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240502 01:55:14.632401 16157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:14.633702 16142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:55:14.633827 16142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cdd4d50250184223992325c86cb6c68e"
format_stamp: "Formatted at 2024-05-02 01:55:14 on dist-test-slave-06l8"
I20240502 01:55:14.633992 16142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:14.666571 16142 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:55:14.667407 16142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:14.667649 16142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:14.668552 16142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:14.670054 16142 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:14.670143 16142 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:14.670264 16142 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:14.670339 16142 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:14.700465 16142 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.193:41953
I20240502 01:55:14.700536 16270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.193:41953 every 8 connection(s)
I20240502 01:55:14.701804 16142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 01:55:14.707867 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 16142
I20240502 01:55:14.708045 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 01:55:14.711097 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.194:0
--local_ip_for_outbound_sockets=127.31.161.194
--webserver_interface=127.31.161.194
--webserver_port=0
--tserver_master_addrs=127.31.161.254:46705
--builtin_ntp_servers=127.31.161.212:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:14.714264 16271 heartbeater.cc:347] Connected to a master server at 127.31.161.254:46705
I20240502 01:55:14.714538 16271 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:14.715308 16271 heartbeater.cc:511] Master 127.31.161.254:46705 requested a full tablet report, sending...
I20240502 01:55:14.762612 16083 ts_manager.cc:194] Registered new tserver with Master: cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953)
I20240502 01:55:14.765028 16083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.193:34461
W20240502 01:55:15.045225 16275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:15.045516 16275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:15.045611 16275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:15.052938 16275 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:15.053061 16275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:15.053218 16275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.194
I20240502 01:55:15.061995 16275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.31.161.194
--webserver_port=0
--tserver_master_addrs=127.31.161.254:46705
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.194
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:15.063231 16275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:15.064867 16275 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:15.076490 16284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:15.077309 16275 server_base.cc:1034] running on GCE node
W20240502 01:55:15.077348 16281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:15.077643 16282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:15.078262 16275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:15.079113 16275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:15.080302 16275 hybrid_clock.cc:648] HybridClock initialized: now 1714614915080256 us; error 36 us; skew 500 ppm
I20240502 01:55:15.080602 16275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:15.082708 16275 webserver.cc:460] Webserver started at http://127.31.161.194:43435/ using document root <none> and password file <none>
I20240502 01:55:15.083422 16275 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:15.083531 16275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:15.083869 16275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:15.086531 16275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
format_stamp: "Formatted at 2024-05-02 01:55:15 on dist-test-slave-06l8"
I20240502 01:55:15.087267 16275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
format_stamp: "Formatted at 2024-05-02 01:55:15 on dist-test-slave-06l8"
I20240502 01:55:15.092424 16275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20240502 01:55:15.095803 16290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:15.097067 16275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240502 01:55:15.097267 16275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
format_stamp: "Formatted at 2024-05-02 01:55:15 on dist-test-slave-06l8"
I20240502 01:55:15.097460 16275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:15.129448 16275 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:55:15.130328 16275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:15.130589 16275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:15.131546 16275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:15.133061 16275 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:15.133172 16275 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:15.133286 16275 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:15.133354 16275 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:15.162829 16275 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.194:38133
I20240502 01:55:15.162874 16403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.194:38133 every 8 connection(s)
I20240502 01:55:15.164379 16275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 01:55:15.164618 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 16275
I20240502 01:55:15.164784 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 01:55:15.168802 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.195:0
--local_ip_for_outbound_sockets=127.31.161.195
--webserver_interface=127.31.161.195
--webserver_port=0
--tserver_master_addrs=127.31.161.254:46705
--builtin_ntp_servers=127.31.161.212:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:15.182687 16404 heartbeater.cc:347] Connected to a master server at 127.31.161.254:46705
I20240502 01:55:15.183004 16404 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:15.183876 16404 heartbeater.cc:511] Master 127.31.161.254:46705 requested a full tablet report, sending...
I20240502 01:55:15.228173 16083 ts_manager.cc:194] Registered new tserver with Master: dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133)
I20240502 01:55:15.229562 16083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.194:35631
W20240502 01:55:15.501475 16407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:15.501802 16407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:15.501905 16407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:15.509173 16407 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:15.509287 16407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:15.509403 16407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.195
I20240502 01:55:15.518226 16407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.31.161.195
--webserver_port=0
--tserver_master_addrs=127.31.161.254:46705
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.195
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:15.519500 16407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:15.521205 16407 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:15.533143 16416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:15.533165 16414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:15.533862 16413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:15.534454 16407 server_base.cc:1034] running on GCE node
I20240502 01:55:15.535004 16407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:15.535918 16407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:15.537180 16407 hybrid_clock.cc:648] HybridClock initialized: now 1714614915537110 us; error 42 us; skew 500 ppm
I20240502 01:55:15.537511 16407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:15.539767 16407 webserver.cc:460] Webserver started at http://127.31.161.195:43753/ using document root <none> and password file <none>
I20240502 01:55:15.540501 16407 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:15.540609 16407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:15.540915 16407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:15.543586 16407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "39059854b64045698ded1ba4e8d47304"
format_stamp: "Formatted at 2024-05-02 01:55:15 on dist-test-slave-06l8"
I20240502 01:55:15.544317 16407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "39059854b64045698ded1ba4e8d47304"
format_stamp: "Formatted at 2024-05-02 01:55:15 on dist-test-slave-06l8"
I20240502 01:55:15.549491 16407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240502 01:55:15.552771 16422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:15.554065 16407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:55:15.554215 16407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "39059854b64045698ded1ba4e8d47304"
format_stamp: "Formatted at 2024-05-02 01:55:15 on dist-test-slave-06l8"
I20240502 01:55:15.554402 16407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:15.591652 16407 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:55:15.592530 16407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:15.592801 16407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:15.593851 16407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:15.595362 16407 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:15.595448 16407 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:15.595554 16407 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:15.595618 16407 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:15.626922 16407 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.195:35693
I20240502 01:55:15.626976 16535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.195:35693 every 8 connection(s)
I20240502 01:55:15.628322 16407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 01:55:15.634080 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 16407
I20240502 01:55:15.634266 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 01:55:15.637571 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.196:0
--local_ip_for_outbound_sockets=127.31.161.196
--webserver_interface=127.31.161.196
--webserver_port=0
--tserver_master_addrs=127.31.161.254:46705
--builtin_ntp_servers=127.31.161.212:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:15.641511 16536 heartbeater.cc:347] Connected to a master server at 127.31.161.254:46705
I20240502 01:55:15.641789 16536 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:15.642591 16536 heartbeater.cc:511] Master 127.31.161.254:46705 requested a full tablet report, sending...
I20240502 01:55:15.686476 16083 ts_manager.cc:194] Registered new tserver with Master: 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:15.687637 16083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.195:39995
I20240502 01:55:15.768599 16271 heartbeater.cc:503] Master 127.31.161.254:46705 was elected leader, sending a full tablet report...
W20240502 01:55:15.968475 16539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:15.968777 16539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:15.968916 16539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:15.976493 16539 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:15.976619 16539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:15.976778 16539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.196
I20240502 01:55:15.986308 16539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.31.161.196
--webserver_port=0
--tserver_master_addrs=127.31.161.254:46705
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.196
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:15.987710 16539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:15.989500 16539 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:16.002367 16546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:16.002911 16548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:16.003067 16545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:16.003634 16539 server_base.cc:1034] running on GCE node
I20240502 01:55:16.004220 16539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:16.005177 16539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:16.006371 16539 hybrid_clock.cc:648] HybridClock initialized: now 1714614916006313 us; error 52 us; skew 500 ppm
I20240502 01:55:16.006685 16539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:16.008901 16539 webserver.cc:460] Webserver started at http://127.31.161.196:36123/ using document root <none> and password file <none>
I20240502 01:55:16.009661 16539 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:16.009773 16539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:16.010078 16539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:16.012656 16539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "703f6f043c6041a4b629933690580367"
format_stamp: "Formatted at 2024-05-02 01:55:16 on dist-test-slave-06l8"
I20240502 01:55:16.013432 16539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "703f6f043c6041a4b629933690580367"
format_stamp: "Formatted at 2024-05-02 01:55:16 on dist-test-slave-06l8"
I20240502 01:55:16.018604 16539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20240502 01:55:16.022094 16554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:16.023391 16539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:55:16.023540 16539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "703f6f043c6041a4b629933690580367"
format_stamp: "Formatted at 2024-05-02 01:55:16 on dist-test-slave-06l8"
I20240502 01:55:16.023725 16539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:16.035660 16539 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:55:16.036489 16539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:16.036751 16539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:16.037870 16539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:16.039496 16539 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:16.039583 16539 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:16.039697 16539 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:16.039764 16539 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:16.068902 16539 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.196:40425
I20240502 01:55:16.068950 16667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.196:40425 every 8 connection(s)
I20240502 01:55:16.070148 16539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 01:55:16.078514 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 16539
I20240502 01:55:16.078688 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 01:55:16.082047 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.197:0
--local_ip_for_outbound_sockets=127.31.161.197
--webserver_interface=127.31.161.197
--webserver_port=0
--tserver_master_addrs=127.31.161.254:46705
--builtin_ntp_servers=127.31.161.212:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:16.082520 16668 heartbeater.cc:347] Connected to a master server at 127.31.161.254:46705
I20240502 01:55:16.082821 16668 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:16.083774 16668 heartbeater.cc:511] Master 127.31.161.254:46705 requested a full tablet report, sending...
I20240502 01:55:16.129735 16083 ts_manager.cc:194] Registered new tserver with Master: 703f6f043c6041a4b629933690580367 (127.31.161.196:40425)
I20240502 01:55:16.131031 16083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.196:38597
I20240502 01:55:16.232349 16404 heartbeater.cc:503] Master 127.31.161.254:46705 was elected leader, sending a full tablet report...
W20240502 01:55:16.411428 16671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:16.411724 16671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:16.411837 16671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:16.419420 16671 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:16.419545 16671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:16.419703 16671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.197
I20240502 01:55:16.428702 16671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.31.161.197
--webserver_port=0
--tserver_master_addrs=127.31.161.254:46705
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.197
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:16.430032 16671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:16.431718 16671 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:16.444036 16677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:16.443984 16678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:16.444015 16680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:16.444756 16671 server_base.cc:1034] running on GCE node
I20240502 01:55:16.445410 16671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:16.446296 16671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:16.447481 16671 hybrid_clock.cc:648] HybridClock initialized: now 1714614916447431 us; error 44 us; skew 500 ppm
I20240502 01:55:16.447793 16671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:16.454960 16671 webserver.cc:460] Webserver started at http://127.31.161.197:38835/ using document root <none> and password file <none>
I20240502 01:55:16.455801 16671 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:16.455942 16671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:16.456272 16671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:16.459028 16671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "ed2e7e4188d0400b8689e28bbf5096ad"
format_stamp: "Formatted at 2024-05-02 01:55:16 on dist-test-slave-06l8"
I20240502 01:55:16.459790 16671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "ed2e7e4188d0400b8689e28bbf5096ad"
format_stamp: "Formatted at 2024-05-02 01:55:16 on dist-test-slave-06l8"
I20240502 01:55:16.465181 16671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20240502 01:55:16.468569 16686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:16.469955 16671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240502 01:55:16.470118 16671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ed2e7e4188d0400b8689e28bbf5096ad"
format_stamp: "Formatted at 2024-05-02 01:55:16 on dist-test-slave-06l8"
I20240502 01:55:16.470327 16671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:16.494309 16671 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:55:16.495184 16671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:16.495496 16671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:16.496613 16671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:16.498291 16671 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:16.498378 16671 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:16.498466 16671 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:16.498519 16671 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:16.532488 16671 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.197:46825
I20240502 01:55:16.532527 16798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.197:46825 every 8 connection(s)
I20240502 01:55:16.533958 16671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 01:55:16.542253 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 16671
I20240502 01:55:16.542416 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 01:55:16.545807 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.198:0
--local_ip_for_outbound_sockets=127.31.161.198
--webserver_interface=127.31.161.198
--webserver_port=0
--tserver_master_addrs=127.31.161.254:46705
--builtin_ntp_servers=127.31.161.212:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:16.546936 16799 heartbeater.cc:347] Connected to a master server at 127.31.161.254:46705
I20240502 01:55:16.547231 16799 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:16.548002 16799 heartbeater.cc:511] Master 127.31.161.254:46705 requested a full tablet report, sending...
I20240502 01:55:16.590690 16083 ts_manager.cc:194] Registered new tserver with Master: ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825)
I20240502 01:55:16.591636 16083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.197:36857
I20240502 01:55:16.690898 16536 heartbeater.cc:503] Master 127.31.161.254:46705 was elected leader, sending a full tablet report...
W20240502 01:55:16.868466 16803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:16.868741 16803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:16.868867 16803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:16.876216 16803 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:16.876324 16803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:16.876467 16803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.198
I20240502 01:55:16.885705 16803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.31.161.198
--webserver_port=0
--tserver_master_addrs=127.31.161.254:46705
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.198
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:16.886924 16803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:16.888638 16803 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:16.899926 16808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:16.900552 16809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:16.900975 16811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:16.901762 16803 server_base.cc:1034] running on GCE node
I20240502 01:55:16.902343 16803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:16.903259 16803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:16.904464 16803 hybrid_clock.cc:648] HybridClock initialized: now 1714614916904419 us; error 39 us; skew 500 ppm
I20240502 01:55:16.904774 16803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:16.906914 16803 webserver.cc:460] Webserver started at http://127.31.161.198:38619/ using document root <none> and password file <none>
I20240502 01:55:16.907610 16803 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:16.907720 16803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:16.908015 16803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:16.910575 16803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "6b9f9aaa1c86420190af75270914449a"
format_stamp: "Formatted at 2024-05-02 01:55:16 on dist-test-slave-06l8"
I20240502 01:55:16.911275 16803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "6b9f9aaa1c86420190af75270914449a"
format_stamp: "Formatted at 2024-05-02 01:55:16 on dist-test-slave-06l8"
I20240502 01:55:16.916152 16803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20240502 01:55:16.919306 16817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:16.920589 16803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:55:16.920735 16803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6b9f9aaa1c86420190af75270914449a"
format_stamp: "Formatted at 2024-05-02 01:55:16 on dist-test-slave-06l8"
I20240502 01:55:16.920912 16803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:16.935932 16803 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:55:16.936769 16803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:16.937031 16803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:16.938050 16803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:16.939544 16803 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:16.939639 16803 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:16.939744 16803 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:16.939812 16803 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:16.978596 16803 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.198:38121
I20240502 01:55:16.978642 16929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.198:38121 every 8 connection(s)
I20240502 01:55:16.979897 16803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 01:55:16.985316 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 16803
I20240502 01:55:16.985486 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 01:55:16.988513 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.199:0
--local_ip_for_outbound_sockets=127.31.161.199
--webserver_interface=127.31.161.199
--webserver_port=0
--tserver_master_addrs=127.31.161.254:46705
--builtin_ntp_servers=127.31.161.212:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:16.992699 16930 heartbeater.cc:347] Connected to a master server at 127.31.161.254:46705
I20240502 01:55:16.992969 16930 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:16.993781 16930 heartbeater.cc:511] Master 127.31.161.254:46705 requested a full tablet report, sending...
I20240502 01:55:17.038386 16083 ts_manager.cc:194] Registered new tserver with Master: 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121)
I20240502 01:55:17.039448 16083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.198:44353
I20240502 01:55:17.134400 16668 heartbeater.cc:503] Master 127.31.161.254:46705 was elected leader, sending a full tablet report...
W20240502 01:55:17.318257 16933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:17.318549 16933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:17.318672 16933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:17.326071 16933 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:17.326213 16933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:17.326340 16933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.199
I20240502 01:55:17.335249 16933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.31.161.199
--webserver_port=0
--tserver_master_addrs=127.31.161.254:46705
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.199
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:17.336524 16933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:17.338322 16933 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:17.349977 16939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:17.350174 16942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:17.350683 16940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:17.351529 16933 server_base.cc:1034] running on GCE node
I20240502 01:55:17.352092 16933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:17.353006 16933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:17.354236 16933 hybrid_clock.cc:648] HybridClock initialized: now 1714614917354175 us; error 37 us; skew 500 ppm
I20240502 01:55:17.354532 16933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:17.356662 16933 webserver.cc:460] Webserver started at http://127.31.161.199:41153/ using document root <none> and password file <none>
I20240502 01:55:17.357414 16933 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:17.357525 16933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:17.357810 16933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:17.360467 16933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "a6575ebf978a4631a13ba5c243c3c4f9"
format_stamp: "Formatted at 2024-05-02 01:55:17 on dist-test-slave-06l8"
I20240502 01:55:17.361189 16933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "a6575ebf978a4631a13ba5c243c3c4f9"
format_stamp: "Formatted at 2024-05-02 01:55:17 on dist-test-slave-06l8"
I20240502 01:55:17.366000 16933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240502 01:55:17.369302 16948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:17.370633 16933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:55:17.370795 16933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "a6575ebf978a4631a13ba5c243c3c4f9"
format_stamp: "Formatted at 2024-05-02 01:55:17 on dist-test-slave-06l8"
I20240502 01:55:17.370985 16933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:17.399310 16933 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:55:17.400171 16933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:17.400431 16933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:17.401417 16933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:17.402927 16933 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:17.403028 16933 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:17.403134 16933 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:17.403201 16933 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:17.431728 16933 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.199:46837
I20240502 01:55:17.431799 17060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.199:46837 every 8 connection(s)
I20240502 01:55:17.432998 16933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 01:55:17.439819 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 16933
I20240502 01:55:17.439970 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 01:55:17.443253 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.200:0
--local_ip_for_outbound_sockets=127.31.161.200
--webserver_interface=127.31.161.200
--webserver_port=0
--tserver_master_addrs=127.31.161.254:46705
--builtin_ntp_servers=127.31.161.212:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:17.445627 17061 heartbeater.cc:347] Connected to a master server at 127.31.161.254:46705
I20240502 01:55:17.445899 17061 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:17.446659 17061 heartbeater.cc:511] Master 127.31.161.254:46705 requested a full tablet report, sending...
I20240502 01:55:17.520959 16083 ts_manager.cc:194] Registered new tserver with Master: a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:17.522377 16083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.199:57685
I20240502 01:55:17.594352 16799 heartbeater.cc:503] Master 127.31.161.254:46705 was elected leader, sending a full tablet report...
W20240502 01:55:17.775696 17064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:17.776021 17064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:17.776175 17064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:17.783764 17064 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:17.783928 17064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:17.784101 17064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.200
I20240502 01:55:17.793543 17064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.31.161.200
--webserver_port=0
--tserver_master_addrs=127.31.161.254:46705
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.200
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:17.794854 17064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:17.796624 17064 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:17.808404 17070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:17.808965 17073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:17.809165 17064 server_base.cc:1034] running on GCE node
W20240502 01:55:17.809682 17071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:17.810312 17064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:17.811182 17064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:17.812381 17064 hybrid_clock.cc:648] HybridClock initialized: now 1714614917812337 us; error 38 us; skew 500 ppm
I20240502 01:55:17.812680 17064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:17.814760 17064 webserver.cc:460] Webserver started at http://127.31.161.200:35675/ using document root <none> and password file <none>
I20240502 01:55:17.815448 17064 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:17.815557 17064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:17.815850 17064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:17.818364 17064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "0982e15b4c74425cad8d7d642eb63316"
format_stamp: "Formatted at 2024-05-02 01:55:17 on dist-test-slave-06l8"
I20240502 01:55:17.819152 17064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "0982e15b4c74425cad8d7d642eb63316"
format_stamp: "Formatted at 2024-05-02 01:55:17 on dist-test-slave-06l8"
I20240502 01:55:17.824120 17064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 01:55:17.827307 17079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:17.828598 17064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20240502 01:55:17.828749 17064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "0982e15b4c74425cad8d7d642eb63316"
format_stamp: "Formatted at 2024-05-02 01:55:17 on dist-test-slave-06l8"
I20240502 01:55:17.828938 17064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:17.856926 17064 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:55:17.857892 17064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:17.858151 17064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:17.859153 17064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:17.861219 17064 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:17.861315 17064 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:17.861423 17064 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:17.861493 17064 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:17.890394 17064 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.200:38559
I20240502 01:55:17.890460 17191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.200:38559 every 8 connection(s)
I20240502 01:55:17.891660 17064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:55:17.901763 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 17064
I20240502 01:55:17.901995 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 01:55:17.905128 17192 heartbeater.cc:347] Connected to a master server at 127.31.161.254:46705
I20240502 01:55:17.905418 17192 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:17.905922 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.201:0
--local_ip_for_outbound_sockets=127.31.161.201
--webserver_interface=127.31.161.201
--webserver_port=0
--tserver_master_addrs=127.31.161.254:46705
--builtin_ntp_servers=127.31.161.212:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:17.906212 17192 heartbeater.cc:511] Master 127.31.161.254:46705 requested a full tablet report, sending...
I20240502 01:55:17.965056 16083 ts_manager.cc:194] Registered new tserver with Master: 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200:38559)
I20240502 01:55:17.966070 16083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.200:46947
I20240502 01:55:18.042130 16930 heartbeater.cc:503] Master 127.31.161.254:46705 was elected leader, sending a full tablet report...
W20240502 01:55:18.254238 17196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:18.254547 17196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:18.254681 17196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:18.262101 17196 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:18.262250 17196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:18.262400 17196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.201
I20240502 01:55:18.271203 17196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.201:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.31.161.201
--webserver_port=0
--tserver_master_addrs=127.31.161.254:46705
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.201
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:18.272509 17196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:18.274295 17196 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:18.286237 17202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:18.286832 17204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:18.286952 17201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:18.287497 17196 server_base.cc:1034] running on GCE node
I20240502 01:55:18.288064 17196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:18.289022 17196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:18.290237 17196 hybrid_clock.cc:648] HybridClock initialized: now 1714614918290190 us; error 45 us; skew 500 ppm
I20240502 01:55:18.290556 17196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:18.292662 17196 webserver.cc:460] Webserver started at http://127.31.161.201:46157/ using document root <none> and password file <none>
I20240502 01:55:18.293390 17196 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:18.293502 17196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:18.293792 17196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:18.296281 17196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data/instance:
uuid: "a7c4573db86a415285d274ca57dde659"
format_stamp: "Formatted at 2024-05-02 01:55:18 on dist-test-slave-06l8"
I20240502 01:55:18.296983 17196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/wal/instance:
uuid: "a7c4573db86a415285d274ca57dde659"
format_stamp: "Formatted at 2024-05-02 01:55:18 on dist-test-slave-06l8"
I20240502 01:55:18.301910 17196 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240502 01:55:18.305141 17210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:18.306320 17196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 01:55:18.306454 17196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "a7c4573db86a415285d274ca57dde659"
format_stamp: "Formatted at 2024-05-02 01:55:18 on dist-test-slave-06l8"
I20240502 01:55:18.306633 17196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:18.326684 17196 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:55:18.327733 17196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:18.328029 17196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:18.329025 17196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:18.330592 17196 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:18.330677 17196 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:18.330785 17196 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:18.330849 17196 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:18.360852 17196 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.201:44737
I20240502 01:55:18.360917 17323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.201:44737 every 8 connection(s)
I20240502 01:55:18.362162 17196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240502 01:55:18.363358 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 17196
I20240502 01:55:18.363529 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/wal/instance
I20240502 01:55:18.367205 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.202:0
--local_ip_for_outbound_sockets=127.31.161.202
--webserver_interface=127.31.161.202
--webserver_port=0
--tserver_master_addrs=127.31.161.254:46705
--builtin_ntp_servers=127.31.161.212:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:18.376577 17324 heartbeater.cc:347] Connected to a master server at 127.31.161.254:46705
I20240502 01:55:18.376886 17324 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:18.377759 17324 heartbeater.cc:511] Master 127.31.161.254:46705 requested a full tablet report, sending...
I20240502 01:55:18.438171 16083 ts_manager.cc:194] Registered new tserver with Master: a7c4573db86a415285d274ca57dde659 (127.31.161.201:44737)
I20240502 01:55:18.439174 16083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.201:34657
I20240502 01:55:18.525835 17061 heartbeater.cc:503] Master 127.31.161.254:46705 was elected leader, sending a full tablet report...
W20240502 01:55:18.720782 17327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:18.721096 17327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:18.721231 17327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:18.728786 17327 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:18.728904 17327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:18.729157 17327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.202
I20240502 01:55:18.739864 17327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.202:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--webserver_interface=127.31.161.202
--webserver_port=0
--tserver_master_addrs=127.31.161.254:46705
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.202
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:18.741151 17327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:18.743166 17327 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:18.754362 17333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:18.754899 17334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:18.755353 17336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:18.755586 17327 server_base.cc:1034] running on GCE node
I20240502 01:55:18.756150 17327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:18.757012 17327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:18.758241 17327 hybrid_clock.cc:648] HybridClock initialized: now 1714614918758192 us; error 51 us; skew 500 ppm
I20240502 01:55:18.758561 17327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:18.760763 17327 webserver.cc:460] Webserver started at http://127.31.161.202:41537/ using document root <none> and password file <none>
I20240502 01:55:18.761504 17327 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:18.761616 17327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:18.761909 17327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 01:55:18.764446 17327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/data/instance:
uuid: "aa7ec9abef1747e5a9e2404cee265c20"
format_stamp: "Formatted at 2024-05-02 01:55:18 on dist-test-slave-06l8"
I20240502 01:55:18.765184 17327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/wal/instance:
uuid: "aa7ec9abef1747e5a9e2404cee265c20"
format_stamp: "Formatted at 2024-05-02 01:55:18 on dist-test-slave-06l8"
I20240502 01:55:18.770097 17327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240502 01:55:18.773553 17342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:18.774876 17327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:55:18.775040 17327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/wal
uuid: "aa7ec9abef1747e5a9e2404cee265c20"
format_stamp: "Formatted at 2024-05-02 01:55:18 on dist-test-slave-06l8"
I20240502 01:55:18.775234 17327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:18.795219 17327 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:55:18.796036 17327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:18.796294 17327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:18.797227 17327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:18.798686 17327 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:18.798770 17327 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:18.798877 17327 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:18.798943 17327 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:18.828197 17327 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.202:43977
I20240502 01:55:18.828251 17454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.202:43977 every 8 connection(s)
I20240502 01:55:18.829505 17327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/data/info.pb
I20240502 01:55:18.834049 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 17327
I20240502 01:55:18.834234 32391 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/wal/instance
I20240502 01:55:18.841753 17455 heartbeater.cc:347] Connected to a master server at 127.31.161.254:46705
I20240502 01:55:18.842025 17455 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:18.842797 17455 heartbeater.cc:511] Master 127.31.161.254:46705 requested a full tablet report, sending...
I20240502 01:55:18.886221 16083 ts_manager.cc:194] Registered new tserver with Master: aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202:43977)
I20240502 01:55:18.887144 16083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.202:35363
I20240502 01:55:18.890132 32391 external_mini_cluster.cc:930] 10 TS(s) registered with all masters
I20240502 01:55:18.906834 32391 external_mini_cluster.cc:930] 10 TS(s) registered with all masters
I20240502 01:55:18.906970 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 17196
I20240502 01:55:18.914722 17319 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:18.968853 17192 heartbeater.cc:503] Master 127.31.161.254:46705 was elected leader, sending a full tablet report...
I20240502 01:55:18.996445 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 17196
I20240502 01:55:19.010803 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 17327
I20240502 01:55:19.019191 17450 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:19.098536 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 17327
I20240502 01:55:19.112563 32391 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMchxvY/build/asan/bin/kudu pid 17064
I20240502 01:55:19.120669 17187 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 01:55:19.201190 32391 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMchxvY/build/asan/bin/kudu with pid 17064
I20240502 01:55:22.967159 16083 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58366:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 01:55:22.973806 16083 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240502 01:55:23.032575 16865 tablet_service.cc:1467] Processing CreateTablet for tablet 48a81a0b7d47479e929d80041f34a3b1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:23.037642 16205 tablet_service.cc:1467] Processing CreateTablet for tablet 48a81a0b7d47479e929d80041f34a3b1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:23.037791 16204 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4e2e002a7d441398f3da0ec650b8d2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:23.039098 16205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a81a0b7d47479e929d80041f34a3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:23.040151 16204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4e2e002a7d441398f3da0ec650b8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:23.041038 16203 tablet_service.cc:1467] Processing CreateTablet for tablet b55b98b06aca47988b14f486973cc370 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:23.041584 16203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55b98b06aca47988b14f486973cc370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:23.044628 16865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a81a0b7d47479e929d80041f34a3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:23.052289 16734 tablet_service.cc:1467] Processing CreateTablet for tablet 48a81a0b7d47479e929d80041f34a3b1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:23.053596 16733 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4e2e002a7d441398f3da0ec650b8d2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:23.054483 16733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4e2e002a7d441398f3da0ec650b8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:23.057535 16732 tablet_service.cc:1467] Processing CreateTablet for tablet b55b98b06aca47988b14f486973cc370 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:23.057945 16732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55b98b06aca47988b14f486973cc370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:23.058914 17487 tablet_bootstrap.cc:492] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e: Bootstrap starting.
I20240502 01:55:23.061487 16734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a81a0b7d47479e929d80041f34a3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:23.061767 17487 tablet_bootstrap.cc:654] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:23.063119 17487 log.cc:824] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:23.067452 16996 tablet_service.cc:1467] Processing CreateTablet for tablet 48a81a0b7d47479e929d80041f34a3b1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:23.068518 16994 tablet_service.cc:1467] Processing CreateTablet for tablet b55b98b06aca47988b14f486973cc370 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:23.070050 16994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55b98b06aca47988b14f486973cc370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:23.068992 16995 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4e2e002a7d441398f3da0ec650b8d2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:23.073566 16995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4e2e002a7d441398f3da0ec650b8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:23.074479 16996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a81a0b7d47479e929d80041f34a3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:23.078207 16338 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4e2e002a7d441398f3da0ec650b8d2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:23.078526 16602 tablet_service.cc:1467] Processing CreateTablet for tablet 48a81a0b7d47479e929d80041f34a3b1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:23.079309 16601 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4e2e002a7d441398f3da0ec650b8d2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:23.079957 16602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a81a0b7d47479e929d80041f34a3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:23.080397 16601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4e2e002a7d441398f3da0ec650b8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:23.082856 16864 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4e2e002a7d441398f3da0ec650b8d2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:23.083321 16864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4e2e002a7d441398f3da0ec650b8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:23.081569 16337 tablet_service.cc:1467] Processing CreateTablet for tablet b55b98b06aca47988b14f486973cc370 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:23.084524 16337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55b98b06aca47988b14f486973cc370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:23.085539 16863 tablet_service.cc:1467] Processing CreateTablet for tablet b55b98b06aca47988b14f486973cc370 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:23.085991 16863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55b98b06aca47988b14f486973cc370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:23.083473 16339 tablet_service.cc:1467] Processing CreateTablet for tablet 48a81a0b7d47479e929d80041f34a3b1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:23.088938 16339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a81a0b7d47479e929d80041f34a3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:23.089409 16600 tablet_service.cc:1467] Processing CreateTablet for tablet b55b98b06aca47988b14f486973cc370 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:23.089927 16600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55b98b06aca47988b14f486973cc370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:23.089991 16338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4e2e002a7d441398f3da0ec650b8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:23.108455 17492 tablet_bootstrap.cc:492] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9: Bootstrap starting.
I20240502 01:55:23.112915 17492 tablet_bootstrap.cc:654] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:23.114257 17490 tablet_bootstrap.cc:492] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:23.115492 17492 log.cc:824] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:23.116560 17489 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: Bootstrap starting.
I20240502 01:55:23.121642 17490 tablet_bootstrap.cc:654] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:23.123092 17489 tablet_bootstrap.cc:654] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:23.123158 16470 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4e2e002a7d441398f3da0ec650b8d2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:23.123888 17487 tablet_bootstrap.cc:492] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e: No bootstrap required, opened a new log
I20240502 01:55:23.124482 17490 log.cc:824] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:23.124650 16470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4e2e002a7d441398f3da0ec650b8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:23.124801 17487 ts_tablet_manager.cc:1367] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e: Time spent bootstrapping tablet: real 0.068s user 0.010s sys 0.005s
I20240502 01:55:23.125535 16471 tablet_service.cc:1467] Processing CreateTablet for tablet 48a81a0b7d47479e929d80041f34a3b1 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:23.125895 17489 log.cc:824] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:23.125998 16471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a81a0b7d47479e929d80041f34a3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:23.127002 17494 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb: Bootstrap starting.
I20240502 01:55:23.127781 17491 tablet_bootstrap.cc:492] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:23.128515 16469 tablet_service.cc:1467] Processing CreateTablet for tablet b55b98b06aca47988b14f486973cc370 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:23.128930 16469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55b98b06aca47988b14f486973cc370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:23.129777 17494 tablet_bootstrap.cc:654] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:23.130936 17491 tablet_bootstrap.cc:654] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:23.130939 17494 log.cc:824] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:23.137735 17487 raft_consensus.cc:348] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:23.138432 17487 raft_consensus.cc:374] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:23.138564 17487 raft_consensus.cc:729] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4d50250184223992325c86cb6c68e, State: Initialized, Role: FOLLOWER
I20240502 01:55:23.139480 17487 consensus_queue.cc:260] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:23.140805 17489 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: No bootstrap required, opened a new log
I20240502 01:55:23.141435 17491 log.cc:824] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:23.141856 17489 ts_tablet_manager.cc:1367] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.000s
I20240502 01:55:23.144063 17490 tablet_bootstrap.cc:492] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad: No bootstrap required, opened a new log
I20240502 01:55:23.144402 17490 ts_tablet_manager.cc:1367] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.000s
I20240502 01:55:23.160285 17491 tablet_bootstrap.cc:492] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367: No bootstrap required, opened a new log
I20240502 01:55:23.160777 17491 ts_tablet_manager.cc:1367] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.007s
I20240502 01:55:23.166958 17494 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb: No bootstrap required, opened a new log
I20240502 01:55:23.167407 17499 tablet_bootstrap.cc:492] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304: Bootstrap starting.
I20240502 01:55:23.170133 17492 tablet_bootstrap.cc:492] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9: No bootstrap required, opened a new log
I20240502 01:55:23.170356 17490 raft_consensus.cc:348] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.170892 17490 raft_consensus.cc:374] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:23.170990 17490 raft_consensus.cc:729] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: FOLLOWER
I20240502 01:55:23.171324 17492 ts_tablet_manager.cc:1367] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent bootstrapping tablet: real 0.063s user 0.011s sys 0.000s
I20240502 01:55:23.171716 17490 consensus_queue.cc:260] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.173053 17494 ts_tablet_manager.cc:1367] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent bootstrapping tablet: real 0.046s user 0.017s sys 0.013s
I20240502 01:55:23.173633 17487 ts_tablet_manager.cc:1398] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e: Time spent starting tablet: real 0.049s user 0.012s sys 0.033s
I20240502 01:55:23.174068 17487 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e: Bootstrap starting.
I20240502 01:55:23.176301 17487 tablet_bootstrap.cc:654] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:23.186194 17489 raft_consensus.cc:348] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.186712 17489 raft_consensus.cc:374] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:23.186818 17489 raft_consensus.cc:729] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Initialized, Role: FOLLOWER
I20240502 01:55:23.187619 17489 consensus_queue.cc:260] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.189006 17491 raft_consensus.cc:348] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.189891 17491 raft_consensus.cc:374] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:23.190042 17491 raft_consensus.cc:729] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: FOLLOWER
I20240502 01:55:23.190383 17499 tablet_bootstrap.cc:654] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:23.190407 17490 ts_tablet_manager.cc:1398] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.046s user 0.011s sys 0.000s
I20240502 01:55:23.190909 17491 consensus_queue.cc:260] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.192993 17501 raft_consensus.cc:482] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:23.193289 17501 raft_consensus.cc:504] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.195313 17499 log.cc:824] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:23.196171 17494 raft_consensus.cc:348] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.197013 17494 raft_consensus.cc:374] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:23.197161 17494 raft_consensus.cc:729] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fcb8d12cb4ab28ec821c9473adeeb, State: Initialized, Role: FOLLOWER
I20240502 01:55:23.198038 17494 consensus_queue.cc:260] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.198160 17492 raft_consensus.cc:348] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:23.198817 17492 raft_consensus.cc:374] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:23.198956 17492 raft_consensus.cc:729] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Initialized, Role: FOLLOWER
I20240502 01:55:23.199828 17492 consensus_queue.cc:260] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:23.193295 17490 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:23.202252 17489 ts_tablet_manager.cc:1398] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: Time spent starting tablet: real 0.055s user 0.011s sys 0.006s
I20240502 01:55:23.203140 17489 tablet_bootstrap.cc:492] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a: Bootstrap starting.
I20240502 01:55:23.203207 17490 tablet_bootstrap.cc:654] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:23.206041 17487 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e: No bootstrap required, opened a new log
I20240502 01:55:23.206274 17487 ts_tablet_manager.cc:1367] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.008s
I20240502 01:55:23.207033 17501 leader_election.cc:288] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:23.208400 17487 raft_consensus.cc:348] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.208726 17487 raft_consensus.cc:374] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:23.208817 17487 raft_consensus.cc:729] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4d50250184223992325c86cb6c68e, State: Initialized, Role: FOLLOWER
I20240502 01:55:23.209183 17487 consensus_queue.cc:260] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
W20240502 01:55:23.209465 16272 tablet.cc:2369] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:55:23.210736 17487 ts_tablet_manager.cc:1398] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240502 01:55:23.211031 17487 tablet_bootstrap.cc:492] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e: Bootstrap starting.
I20240502 01:55:23.212836 17487 tablet_bootstrap.cc:654] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:23.213503 17489 tablet_bootstrap.cc:654] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:23.218184 17499 tablet_bootstrap.cc:492] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304: No bootstrap required, opened a new log
I20240502 01:55:23.218530 17499 ts_tablet_manager.cc:1367] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.004s
I20240502 01:55:23.225417 17489 tablet_bootstrap.cc:492] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a: No bootstrap required, opened a new log
I20240502 01:55:23.225692 17489 ts_tablet_manager.cc:1367] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20240502 01:55:23.226959 17489 raft_consensus.cc:348] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.227352 17489 raft_consensus.cc:374] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:23.227478 17489 raft_consensus.cc:729] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Initialized, Role: FOLLOWER
I20240502 01:55:23.227905 17489 consensus_queue.cc:260] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.232115 17492 ts_tablet_manager.cc:1398] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent starting tablet: real 0.061s user 0.008s sys 0.004s
I20240502 01:55:23.232599 17492 tablet_bootstrap.cc:492] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9: Bootstrap starting.
I20240502 01:55:23.236394 17492 tablet_bootstrap.cc:654] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:23.239640 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4e2e002a7d441398f3da0ec650b8d2" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:23.239965 17490 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad: No bootstrap required, opened a new log
I20240502 01:55:23.240134 17490 ts_tablet_manager.cc:1367] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.047s user 0.012s sys 0.004s
I20240502 01:55:23.240199 16622 raft_consensus.cc:2454] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:23.241196 17490 raft_consensus.cc:348] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.241515 17491 ts_tablet_manager.cc:1398] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.081s user 0.015s sys 0.003s
I20240502 01:55:23.241974 17491 tablet_bootstrap.cc:492] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:23.242579 17492 tablet_bootstrap.cc:492] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9: No bootstrap required, opened a new log
I20240502 01:55:23.242815 17492 ts_tablet_manager.cc:1367] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 01:55:23.242605 17499 raft_consensus.cc:348] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:23.243188 17499 raft_consensus.cc:374] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:23.243335 17499 raft_consensus.cc:729] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Initialized, Role: FOLLOWER
I20240502 01:55:23.244093 17492 raft_consensus.cc:348] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.244200 17499 consensus_queue.cc:260] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:23.244462 17490 raft_consensus.cc:374] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:23.244652 17490 raft_consensus.cc:729] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: FOLLOWER
I20240502 01:55:23.244804 17492 raft_consensus.cc:374] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:23.245069 17492 raft_consensus.cc:729] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Initialized, Role: FOLLOWER
I20240502 01:55:23.245360 17490 consensus_queue.cc:260] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.245982 17492 consensus_queue.cc:260] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.246745 17490 ts_tablet_manager.cc:1398] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 01:55:23.247176 17492 ts_tablet_manager.cc:1398] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:55:23.247524 17490 tablet_bootstrap.cc:492] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:23.247867 17492 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9: Bootstrap starting.
I20240502 01:55:23.251075 17492 tablet_bootstrap.cc:654] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:23.253093 17490 tablet_bootstrap.cc:654] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:23.255440 17491 tablet_bootstrap.cc:654] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367: Neither blocks nor log segments found. Creating new log.
W20240502 01:55:23.235503 16931 tablet.cc:2369] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:55:23.257452 17487 tablet_bootstrap.cc:492] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e: No bootstrap required, opened a new log
I20240502 01:55:23.257481 17489 ts_tablet_manager.cc:1398] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a: Time spent starting tablet: real 0.032s user 0.000s sys 0.002s
I20240502 01:55:23.257740 17487 ts_tablet_manager.cc:1367] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e: Time spent bootstrapping tablet: real 0.047s user 0.001s sys 0.003s
I20240502 01:55:23.257786 17489 tablet_bootstrap.cc:492] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a: Bootstrap starting.
I20240502 01:55:23.258666 17487 raft_consensus.cc:348] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.259043 17487 raft_consensus.cc:374] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:23.259167 17487 raft_consensus.cc:729] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4d50250184223992325c86cb6c68e, State: Initialized, Role: FOLLOWER
I20240502 01:55:23.259601 17487 consensus_queue.cc:260] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.260389 17487 ts_tablet_manager.cc:1398] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:23.265602 17490 tablet_bootstrap.cc:492] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad: No bootstrap required, opened a new log
I20240502 01:55:23.265889 17490 ts_tablet_manager.cc:1367] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.005s
I20240502 01:55:23.266724 17490 raft_consensus.cc:348] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:23.267678 17490 raft_consensus.cc:374] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:23.267810 17490 raft_consensus.cc:729] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: FOLLOWER
I20240502 01:55:23.267804 17494 ts_tablet_manager.cc:1398] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent starting tablet: real 0.095s user 0.014s sys 0.010s
I20240502 01:55:23.268144 17490 consensus_queue.cc:260] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:23.269349 17494 tablet_bootstrap.cc:492] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb: Bootstrap starting.
I20240502 01:55:23.269165 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4e2e002a7d441398f3da0ec650b8d2" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:23.270627 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4e2e002a7d441398f3da0ec650b8d2" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:23.271173 16885 raft_consensus.cc:2454] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:23.271626 17489 tablet_bootstrap.cc:654] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:23.272279 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4e2e002a7d441398f3da0ec650b8d2" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:23.272789 16225 raft_consensus.cc:2454] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:23.273521 16491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4e2e002a7d441398f3da0ec650b8d2" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:23.273983 17494 tablet_bootstrap.cc:654] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:23.274237 17499 ts_tablet_manager.cc:1398] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304: Time spent starting tablet: real 0.056s user 0.015s sys 0.012s
I20240502 01:55:23.274775 17499 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304: Bootstrap starting.
I20240502 01:55:23.276571 17492 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9: No bootstrap required, opened a new log
I20240502 01:55:23.276772 17492 ts_tablet_manager.cc:1367] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.003s
I20240502 01:55:23.277495 17499 tablet_bootstrap.cc:654] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:23.277557 17492 raft_consensus.cc:348] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.277890 17492 raft_consensus.cc:374] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:23.277971 17492 raft_consensus.cc:729] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Initialized, Role: FOLLOWER
I20240502 01:55:23.278275 17492 consensus_queue.cc:260] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.278926 17492 ts_tablet_manager.cc:1398] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:55:23.269770 17490 ts_tablet_manager.cc:1398] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:55:23.279273 17491 tablet_bootstrap.cc:492] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367: No bootstrap required, opened a new log
W20240502 01:55:23.271867 16690 leader_election.cc:341] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:23.279444 17491 ts_tablet_manager.cc:1367] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.002s
W20240502 01:55:23.280692 16687 leader_election.cc:341] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:23.280949 16689 leader_election.cc:302] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 6b9f9aaa1c86420190af75270914449a, 703f6f043c6041a4b629933690580367, cdd4d50250184223992325c86cb6c68e, ed2e7e4188d0400b8689e28bbf5096ad; no voters: 39059854b64045698ded1ba4e8d47304, dc9fcb8d12cb4ab28ec821c9473adeeb
I20240502 01:55:23.281579 17501 raft_consensus.cc:2781] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:23.281729 17501 raft_consensus.cc:482] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:23.281824 17501 raft_consensus.cc:3037] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:23.284711 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4e2e002a7d441398f3da0ec650b8d2" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:23.284858 17501 raft_consensus.cc:504] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.285257 17016 raft_consensus.cc:2454] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:23.286455 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4e2e002a7d441398f3da0ec650b8d2" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367"
I20240502 01:55:23.286677 16622 raft_consensus.cc:3037] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:23.288879 16622 raft_consensus.cc:2454] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:23.289310 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4e2e002a7d441398f3da0ec650b8d2" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a"
I20240502 01:55:23.289541 16885 raft_consensus.cc:3037] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:55:23.290254 16800 tablet.cc:2369] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:55:23.291952 16885 raft_consensus.cc:2454] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:23.292872 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4e2e002a7d441398f3da0ec650b8d2" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9"
I20240502 01:55:23.292969 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4e2e002a7d441398f3da0ec650b8d2" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e"
I20240502 01:55:23.293061 17016 raft_consensus.cc:3037] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:23.293200 16225 raft_consensus.cc:3037] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:23.296011 17016 raft_consensus.cc:2454] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:23.296011 16225 raft_consensus.cc:2454] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:23.296452 16689 leader_election.cc:302] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 6b9f9aaa1c86420190af75270914449a, 703f6f043c6041a4b629933690580367, cdd4d50250184223992325c86cb6c68e, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:23.297137 17501 leader_election.cc:288] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 election: Requested vote from peers 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:23.297339 17501 raft_consensus.cc:2781] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:23.297467 17501 raft_consensus.cc:686] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Becoming Leader. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Running, Role: LEADER
I20240502 01:55:23.297595 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4e2e002a7d441398f3da0ec650b8d2" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
I20240502 01:55:23.298030 17501 consensus_queue.cc:237] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
W20240502 01:55:23.298318 16690 leader_election.cc:341] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:23.299139 16491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4e2e002a7d441398f3da0ec650b8d2" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304"
W20240502 01:55:23.299801 16687 leader_election.cc:341] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:23.301716 17491 raft_consensus.cc:348] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:23.302006 17491 raft_consensus.cc:374] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:23.302119 17491 raft_consensus.cc:729] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: FOLLOWER
I20240502 01:55:23.302450 17491 consensus_queue.cc:260] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:23.303179 17491 ts_tablet_manager.cc:1398] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.024s user 0.002s sys 0.001s
I20240502 01:55:23.303505 17491 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:23.304994 17491 tablet_bootstrap.cc:654] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:23.308055 17489 tablet_bootstrap.cc:492] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a: No bootstrap required, opened a new log
I20240502 01:55:23.308261 17489 ts_tablet_manager.cc:1367] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.012s
I20240502 01:55:23.308441 17491 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367: No bootstrap required, opened a new log
I20240502 01:55:23.308498 17499 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304: No bootstrap required, opened a new log
I20240502 01:55:23.308607 17491 ts_tablet_manager.cc:1367] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:55:23.308640 17499 ts_tablet_manager.cc:1367] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.018s
I20240502 01:55:23.309015 17489 raft_consensus.cc:348] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:23.309352 17489 raft_consensus.cc:374] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:23.309432 17489 raft_consensus.cc:729] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Initialized, Role: FOLLOWER
I20240502 01:55:23.308504 16081 catalog_manager.cc:5484] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: term changed from 0 to 1, leader changed from <none> to ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197). New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:23.309581 17499 raft_consensus.cc:348] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.309857 17499 raft_consensus.cc:374] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:23.309705 17489 consensus_queue.cc:260] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:23.309952 17499 raft_consensus.cc:729] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Initialized, Role: FOLLOWER
I20240502 01:55:23.309723 17491 raft_consensus.cc:348] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.310226 17491 raft_consensus.cc:374] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:23.310343 17491 raft_consensus.cc:729] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: FOLLOWER
I20240502 01:55:23.311038 17494 tablet_bootstrap.cc:492] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb: No bootstrap required, opened a new log
I20240502 01:55:23.310853 17491 consensus_queue.cc:260] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.311213 17494 ts_tablet_manager.cc:1367] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent bootstrapping tablet: real 0.043s user 0.007s sys 0.011s
I20240502 01:55:23.311869 17489 ts_tablet_manager.cc:1398] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240502 01:55:23.312355 17491 ts_tablet_manager.cc:1398] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:55:23.312207 17494 raft_consensus.cc:348] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.312484 17494 raft_consensus.cc:374] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:23.312583 17494 raft_consensus.cc:729] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fcb8d12cb4ab28ec821c9473adeeb, State: Initialized, Role: FOLLOWER
I20240502 01:55:23.312805 17494 consensus_queue.cc:260] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.313324 17494 ts_tablet_manager.cc:1398] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 01:55:23.313627 17494 tablet_bootstrap.cc:492] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb: Bootstrap starting.
I20240502 01:55:23.314643 17499 consensus_queue.cc:260] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.315253 17494 tablet_bootstrap.cc:654] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:23.316799 17499 ts_tablet_manager.cc:1398] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240502 01:55:23.317188 17499 tablet_bootstrap.cc:492] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304: Bootstrap starting.
I20240502 01:55:23.318838 17494 tablet_bootstrap.cc:492] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb: No bootstrap required, opened a new log
I20240502 01:55:23.318888 17499 tablet_bootstrap.cc:654] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:23.318991 17494 ts_tablet_manager.cc:1367] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240502 01:55:23.319567 17494 raft_consensus.cc:348] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:23.319814 17494 raft_consensus.cc:374] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:23.319892 17494 raft_consensus.cc:729] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fcb8d12cb4ab28ec821c9473adeeb, State: Initialized, Role: FOLLOWER
I20240502 01:55:23.320168 17494 consensus_queue.cc:260] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:23.320887 17494 ts_tablet_manager.cc:1398] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:55:23.321029 17499 tablet_bootstrap.cc:492] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304: No bootstrap required, opened a new log
I20240502 01:55:23.321213 17499 ts_tablet_manager.cc:1367] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 01:55:23.321794 17499 raft_consensus.cc:348] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.321998 17499 raft_consensus.cc:374] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:23.322063 17499 raft_consensus.cc:729] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Initialized, Role: FOLLOWER
I20240502 01:55:23.322297 17499 consensus_queue.cc:260] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.322811 17499 ts_tablet_manager.cc:1398] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 01:55:23.327167 16669 tablet.cc:2369] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:55:23.354823 17501 raft_consensus.cc:482] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:23.355064 17501 raft_consensus.cc:504] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:23.356288 17501 leader_election.cc:288] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:23.356560 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55b98b06aca47988b14f486973cc370" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:23.356729 16491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55b98b06aca47988b14f486973cc370" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:23.356868 16225 raft_consensus.cc:2454] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:23.356956 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55b98b06aca47988b14f486973cc370" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:23.357000 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55b98b06aca47988b14f486973cc370" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:23.357134 16491 raft_consensus.cc:2454] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:23.357275 17016 raft_consensus.cc:2454] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:23.357331 16359 raft_consensus.cc:2454] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:23.357712 16687 leader_election.cc:302] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, a6575ebf978a4631a13ba5c243c3c4f9, cdd4d50250184223992325c86cb6c68e, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:23.357916 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55b98b06aca47988b14f486973cc370" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:23.358127 17501 raft_consensus.cc:2781] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:23.358170 16885 raft_consensus.cc:2454] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:23.358242 17501 raft_consensus.cc:482] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:23.358361 17501 raft_consensus.cc:3037] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:23.358405 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55b98b06aca47988b14f486973cc370" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:23.358640 16622 raft_consensus.cc:2454] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:23.360584 17501 raft_consensus.cc:504] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:23.361665 17501 leader_election.cc:288] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 election: Requested vote from peers cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:23.361881 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55b98b06aca47988b14f486973cc370" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
I20240502 01:55:23.361982 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55b98b06aca47988b14f486973cc370" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e"
I20240502 01:55:23.362044 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55b98b06aca47988b14f486973cc370" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a"
I20240502 01:55:23.362083 16359 raft_consensus.cc:3037] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:23.362211 16225 raft_consensus.cc:3037] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:23.362282 16885 raft_consensus.cc:3037] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:23.362097 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55b98b06aca47988b14f486973cc370" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367"
I20240502 01:55:23.362520 16622 raft_consensus.cc:3037] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:23.364948 16225 raft_consensus.cc:2454] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:23.364979 16885 raft_consensus.cc:2454] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:23.365020 16359 raft_consensus.cc:2454] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:23.365125 16622 raft_consensus.cc:2454] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:23.365617 16689 leader_election.cc:302] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 6b9f9aaa1c86420190af75270914449a, 703f6f043c6041a4b629933690580367, cdd4d50250184223992325c86cb6c68e, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:23.365603 16491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55b98b06aca47988b14f486973cc370" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304"
I20240502 01:55:23.365816 16491 raft_consensus.cc:3037] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:23.365829 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55b98b06aca47988b14f486973cc370" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9"
I20240502 01:55:23.365881 17501 raft_consensus.cc:2781] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:23.366040 17016 raft_consensus.cc:3037] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:23.366185 17501 raft_consensus.cc:686] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Becoming Leader. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Running, Role: LEADER
I20240502 01:55:23.366576 17501 consensus_queue.cc:237] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:23.368737 17016 raft_consensus.cc:2454] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:23.368752 16491 raft_consensus.cc:2454] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:23.370098 16083 catalog_manager.cc:5484] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: term changed from 0 to 1, leader changed from <none> to ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197). New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:23.378103 17517 raft_consensus.cc:482] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:23.378262 17517 raft_consensus.cc:504] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.380476 17517 leader_election.cc:288] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133)
W20240502 01:55:23.385066 16537 tablet.cc:2369] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:55:23.393395 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a81a0b7d47479e929d80041f34a3b1" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:23.393651 16885 raft_consensus.cc:2454] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:23.397708 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a81a0b7d47479e929d80041f34a3b1" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:23.398171 16754 raft_consensus.cc:2454] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:23.401645 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a81a0b7d47479e929d80041f34a3b1" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:23.401888 16622 raft_consensus.cc:2454] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:23.402458 16426 leader_election.cc:302] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 6b9f9aaa1c86420190af75270914449a, 703f6f043c6041a4b629933690580367, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:23.402691 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a81a0b7d47479e929d80041f34a3b1" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:23.402926 17016 raft_consensus.cc:2454] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:23.403123 17517 raft_consensus.cc:2781] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:23.403250 17517 raft_consensus.cc:482] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:23.403340 17517 raft_consensus.cc:3037] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:23.403528 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a81a0b7d47479e929d80041f34a3b1" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:23.403793 16359 raft_consensus.cc:2454] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:23.405556 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a81a0b7d47479e929d80041f34a3b1" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:23.405788 16225 raft_consensus.cc:2454] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:23.406131 17517 raft_consensus.cc:504] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.407198 17517 leader_election.cc:288] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 election: Requested vote from peers 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133)
I20240502 01:55:23.407311 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a81a0b7d47479e929d80041f34a3b1" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a"
I20240502 01:55:23.407344 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a81a0b7d47479e929d80041f34a3b1" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad"
I20240502 01:55:23.407518 16885 raft_consensus.cc:3037] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:23.407547 16754 raft_consensus.cc:3037] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:23.407797 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a81a0b7d47479e929d80041f34a3b1" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367"
I20240502 01:55:23.408005 16622 raft_consensus.cc:3037] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:23.409588 16754 raft_consensus.cc:2454] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:23.410125 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a81a0b7d47479e929d80041f34a3b1" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9"
I20240502 01:55:23.410248 16885 raft_consensus.cc:2454] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:23.410305 17016 raft_consensus.cc:3037] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:23.410629 16622 raft_consensus.cc:2454] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:23.410645 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a81a0b7d47479e929d80041f34a3b1" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e"
I20240502 01:55:23.410876 16225 raft_consensus.cc:3037] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:23.411325 16424 leader_election.cc:302] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 6b9f9aaa1c86420190af75270914449a, 703f6f043c6041a4b629933690580367, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:23.411471 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a81a0b7d47479e929d80041f34a3b1" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
I20240502 01:55:23.411623 17517 raft_consensus.cc:2781] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:23.411680 16359 raft_consensus.cc:3037] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:23.412801 17016 raft_consensus.cc:2454] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:23.413779 16225 raft_consensus.cc:2454] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:23.413792 17517 raft_consensus.cc:686] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Becoming Leader. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Running, Role: LEADER
I20240502 01:55:23.414263 16359 raft_consensus.cc:2454] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:23.414439 17517 consensus_queue.cc:237] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.419934 16083 catalog_manager.cc:5484] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39059854b64045698ded1ba4e8d47304 (127.31.161.195). New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } }
W20240502 01:55:23.427878 16405 tablet.cc:2369] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 01:55:23.438278 17062 tablet.cc:2369] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 01:55:23.687523 16359 raft_consensus.cc:3037] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:23.690995 17501 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:23.719040 17517 raft_consensus.cc:482] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:23.719283 17517 raft_consensus.cc:504] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:23.720495 17517 leader_election.cc:288] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133)
I20240502 01:55:23.720818 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4e2e002a7d441398f3da0ec650b8d2" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:23.721199 16622 raft_consensus.cc:2379] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39059854b64045698ded1ba4e8d47304 in current term 1: Already voted for candidate ed2e7e4188d0400b8689e28bbf5096ad in this term.
I20240502 01:55:23.721493 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4e2e002a7d441398f3da0ec650b8d2" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:23.721660 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4e2e002a7d441398f3da0ec650b8d2" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:23.721832 16225 raft_consensus.cc:2379] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39059854b64045698ded1ba4e8d47304 in current term 1: Already voted for candidate ed2e7e4188d0400b8689e28bbf5096ad in this term.
I20240502 01:55:23.721834 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4e2e002a7d441398f3da0ec650b8d2" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:23.721966 16885 raft_consensus.cc:2379] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39059854b64045698ded1ba4e8d47304 in current term 1: Already voted for candidate ed2e7e4188d0400b8689e28bbf5096ad in this term.
I20240502 01:55:23.722177 17016 raft_consensus.cc:2379] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39059854b64045698ded1ba4e8d47304 in current term 1: Already voted for candidate ed2e7e4188d0400b8689e28bbf5096ad in this term.
I20240502 01:55:23.722754 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4e2e002a7d441398f3da0ec650b8d2" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:23.723268 16426 leader_election.cc:302] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 39059854b64045698ded1ba4e8d47304; no voters: 6b9f9aaa1c86420190af75270914449a, 703f6f043c6041a4b629933690580367, a6575ebf978a4631a13ba5c243c3c4f9, cdd4d50250184223992325c86cb6c68e
I20240502 01:55:23.723627 17517 raft_consensus.cc:3037] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:23.723557 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4e2e002a7d441398f3da0ec650b8d2" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:23.726047 17517 raft_consensus.cc:2726] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:55:23.777724 17501 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:23.781759 17501 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:23.785573 17501 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:23.786036 17501 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:23.803759 17517 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:23.806281 17525 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:23.812197 17525 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:23.812656 17525 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:23.848824 17501 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:23.851925 17517 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:23.855593 17525 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:23.856405 17501 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:23.858700 17501 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:23.873170 17538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:23.873608 17538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:23.881630 17538 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 01:55:23.930804 17517 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:23.945514 17589 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:23.945986 17517 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:23.960254 17517 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:24.153627 16690 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:38133, user_credentials={real_user=slave}} blocked reactor thread for 64530.9us
W20240502 01:55:24.245656 16687 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.195:35693, user_credentials={real_user=slave}} blocked reactor thread for 76122.8us
INSERT report
rows total: 2000
time total: 523.469 ms
time per row: 0.261735 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240502 01:55:25.431303 16080 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58366:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 01:55:25.432034 16080 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240502 01:55:25.442917 16733 tablet_service.cc:1467] Processing CreateTablet for tablet 297561b66dd7472ab817c2a23723565c (DEFAULT_TABLE table=rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:25.443399 16733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 297561b66dd7472ab817c2a23723565c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.443462 16865 tablet_service.cc:1467] Processing CreateTablet for tablet 297561b66dd7472ab817c2a23723565c (DEFAULT_TABLE table=rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:25.443885 16865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 297561b66dd7472ab817c2a23723565c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.445204 16339 tablet_service.cc:1467] Processing CreateTablet for tablet f53b466fba6a4655a63a559b0ea56f77 (DEFAULT_TABLE table=rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:25.445626 16339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53b466fba6a4655a63a559b0ea56f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.445564 16204 tablet_service.cc:1467] Processing CreateTablet for tablet 297561b66dd7472ab817c2a23723565c (DEFAULT_TABLE table=rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:25.445997 16204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 297561b66dd7472ab817c2a23723565c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.447216 16994 tablet_service.cc:1467] Processing CreateTablet for tablet 297561b66dd7472ab817c2a23723565c (DEFAULT_TABLE table=rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:25.447620 16994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 297561b66dd7472ab817c2a23723565c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.449513 16601 tablet_service.cc:1467] Processing CreateTablet for tablet 297561b66dd7472ab817c2a23723565c (DEFAULT_TABLE table=rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:25.449923 16601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 297561b66dd7472ab817c2a23723565c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.449826 16338 tablet_service.cc:1467] Processing CreateTablet for tablet 297561b66dd7472ab817c2a23723565c (DEFAULT_TABLE table=rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:25.450249 16338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 297561b66dd7472ab817c2a23723565c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.451877 16205 tablet_service.cc:1467] Processing CreateTablet for tablet f53b466fba6a4655a63a559b0ea56f77 (DEFAULT_TABLE table=rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:25.452289 16205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53b466fba6a4655a63a559b0ea56f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.452656 16469 tablet_service.cc:1467] Processing CreateTablet for tablet 297561b66dd7472ab817c2a23723565c (DEFAULT_TABLE table=rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:25.453666 16863 tablet_service.cc:1467] Processing CreateTablet for tablet f53b466fba6a4655a63a559b0ea56f77 (DEFAULT_TABLE table=rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:25.454044 16863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53b466fba6a4655a63a559b0ea56f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.457415 16996 tablet_service.cc:1467] Processing CreateTablet for tablet f53b466fba6a4655a63a559b0ea56f77 (DEFAULT_TABLE table=rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:25.457856 16996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53b466fba6a4655a63a559b0ea56f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.458245 16602 tablet_service.cc:1467] Processing CreateTablet for tablet f53b466fba6a4655a63a559b0ea56f77 (DEFAULT_TABLE table=rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:25.458622 16602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53b466fba6a4655a63a559b0ea56f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.461441 16600 tablet_service.cc:1467] Processing CreateTablet for tablet 512eab02225d4544a342811f0a7359aa (DEFAULT_TABLE table=rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:25.461802 16600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512eab02225d4544a342811f0a7359aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.462108 16732 tablet_service.cc:1467] Processing CreateTablet for tablet f53b466fba6a4655a63a559b0ea56f77 (DEFAULT_TABLE table=rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:25.462592 16732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53b466fba6a4655a63a559b0ea56f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.465567 16734 tablet_service.cc:1467] Processing CreateTablet for tablet 512eab02225d4544a342811f0a7359aa (DEFAULT_TABLE table=rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:25.465595 16337 tablet_service.cc:1467] Processing CreateTablet for tablet 512eab02225d4544a342811f0a7359aa (DEFAULT_TABLE table=rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:25.466037 16337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512eab02225d4544a342811f0a7359aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.467736 16995 tablet_service.cc:1467] Processing CreateTablet for tablet 512eab02225d4544a342811f0a7359aa (DEFAULT_TABLE table=rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:25.468289 16995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512eab02225d4544a342811f0a7359aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.469302 16734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512eab02225d4544a342811f0a7359aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.473218 16864 tablet_service.cc:1467] Processing CreateTablet for tablet 512eab02225d4544a342811f0a7359aa (DEFAULT_TABLE table=rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:25.473645 16864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512eab02225d4544a342811f0a7359aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.473666 16204 tablet_service.cc:1467] Processing CreateTablet for tablet 512eab02225d4544a342811f0a7359aa (DEFAULT_TABLE table=rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:25.474123 16204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512eab02225d4544a342811f0a7359aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.477222 16471 tablet_service.cc:1467] Processing CreateTablet for tablet f53b466fba6a4655a63a559b0ea56f77 (DEFAULT_TABLE table=rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:25.478245 16469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 297561b66dd7472ab817c2a23723565c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.478688 16470 tablet_service.cc:1467] Processing CreateTablet for tablet 512eab02225d4544a342811f0a7359aa (DEFAULT_TABLE table=rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:25.479035 16470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512eab02225d4544a342811f0a7359aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.481271 16471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53b466fba6a4655a63a559b0ea56f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:25.498220 17654 tablet_bootstrap.cc:492] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9: Bootstrap starting.
I20240502 01:55:25.500015 17654 tablet_bootstrap.cc:654] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:25.518508 17657 tablet_bootstrap.cc:492] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:25.520334 17651 tablet_bootstrap.cc:492] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb: Bootstrap starting.
I20240502 01:55:25.520390 17659 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304: Bootstrap starting.
I20240502 01:55:25.521044 17652 tablet_bootstrap.cc:492] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e: Bootstrap starting.
I20240502 01:55:25.522289 17659 tablet_bootstrap.cc:654] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:25.522805 17652 tablet_bootstrap.cc:654] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:25.523664 17657 tablet_bootstrap.cc:654] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:25.524399 17659 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304: No bootstrap required, opened a new log
I20240502 01:55:25.524895 17659 ts_tablet_manager.cc:1367] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 01:55:25.525717 17659 raft_consensus.cc:348] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.527379 17659 raft_consensus.cc:374] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:25.527504 17659 raft_consensus.cc:729] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Initialized, Role: FOLLOWER
I20240502 01:55:25.527765 17659 consensus_queue.cc:260] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.528658 17656 tablet_bootstrap.cc:492] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:25.528724 17657 tablet_bootstrap.cc:492] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367: No bootstrap required, opened a new log
I20240502 01:55:25.526026 17651 tablet_bootstrap.cc:654] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:25.529292 17659 ts_tablet_manager.cc:1398] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:55:25.530130 17659 tablet_bootstrap.cc:492] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304: Bootstrap starting.
I20240502 01:55:25.531647 17659 tablet_bootstrap.cc:654] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:25.531716 17657 ts_tablet_manager.cc:1367] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240502 01:55:25.532621 17654 tablet_bootstrap.cc:492] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9: No bootstrap required, opened a new log
I20240502 01:55:25.532842 17654 ts_tablet_manager.cc:1367] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.012s
I20240502 01:55:25.532778 17657 raft_consensus.cc:348] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:25.533236 17657 raft_consensus.cc:374] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:25.533386 17657 raft_consensus.cc:729] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: FOLLOWER
I20240502 01:55:25.533651 17654 raft_consensus.cc:348] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.533957 17654 raft_consensus.cc:374] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:25.533728 17657 consensus_queue.cc:260] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:25.534056 17654 raft_consensus.cc:729] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Initialized, Role: FOLLOWER
I20240502 01:55:25.534204 17653 tablet_bootstrap.cc:492] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a: Bootstrap starting.
I20240502 01:55:25.534345 17654 consensus_queue.cc:260] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.535607 17651 tablet_bootstrap.cc:492] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb: No bootstrap required, opened a new log
I20240502 01:55:25.535790 17651 ts_tablet_manager.cc:1367] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.002s
I20240502 01:55:25.536155 17654 ts_tablet_manager.cc:1398] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:55:25.536510 17654 tablet_bootstrap.cc:492] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9: Bootstrap starting.
I20240502 01:55:25.536633 17651 raft_consensus.cc:348] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:25.536928 17651 raft_consensus.cc:374] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:25.537026 17651 raft_consensus.cc:729] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fcb8d12cb4ab28ec821c9473adeeb, State: Initialized, Role: FOLLOWER
I20240502 01:55:25.537672 17651 consensus_queue.cc:260] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:25.538133 17654 tablet_bootstrap.cc:654] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:25.539593 17651 ts_tablet_manager.cc:1398] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:55:25.540762 17651 tablet_bootstrap.cc:492] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb: Bootstrap starting.
I20240502 01:55:25.542373 17651 tablet_bootstrap.cc:654] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:25.545146 17653 tablet_bootstrap.cc:654] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:25.545707 17652 tablet_bootstrap.cc:492] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e: No bootstrap required, opened a new log
I20240502 01:55:25.545939 17652 ts_tablet_manager.cc:1367] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.005s
I20240502 01:55:25.546655 17656 tablet_bootstrap.cc:654] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:25.546855 17652 raft_consensus.cc:348] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.547188 17652 raft_consensus.cc:374] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:25.547698 17652 raft_consensus.cc:729] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4d50250184223992325c86cb6c68e, State: Initialized, Role: FOLLOWER
I20240502 01:55:25.548082 17652 consensus_queue.cc:260] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.549713 17657 ts_tablet_manager.cc:1398] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.018s user 0.004s sys 0.000s
I20240502 01:55:25.550119 17657 tablet_bootstrap.cc:492] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:25.551775 17657 tablet_bootstrap.cc:654] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:25.555212 17653 tablet_bootstrap.cc:492] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a: No bootstrap required, opened a new log
I20240502 01:55:25.555466 17653 ts_tablet_manager.cc:1367] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.006s
I20240502 01:55:25.556118 17659 tablet_bootstrap.cc:492] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304: No bootstrap required, opened a new log
I20240502 01:55:25.556361 17659 ts_tablet_manager.cc:1367] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.007s
I20240502 01:55:25.556419 17653 raft_consensus.cc:348] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.556778 17653 raft_consensus.cc:374] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:25.556913 17653 raft_consensus.cc:729] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Initialized, Role: FOLLOWER
I20240502 01:55:25.557183 17659 raft_consensus.cc:348] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:25.557296 17653 consensus_queue.cc:260] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.558460 17654 tablet_bootstrap.cc:492] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9: No bootstrap required, opened a new log
I20240502 01:55:25.558616 17659 raft_consensus.cc:374] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:25.558743 17659 raft_consensus.cc:729] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Initialized, Role: FOLLOWER
I20240502 01:55:25.559139 17651 tablet_bootstrap.cc:492] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb: No bootstrap required, opened a new log
I20240502 01:55:25.559329 17651 ts_tablet_manager.cc:1367] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240502 01:55:25.559469 17654 ts_tablet_manager.cc:1367] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.006s
I20240502 01:55:25.559088 17659 consensus_queue.cc:260] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:25.560732 17659 ts_tablet_manager.cc:1398] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:55:25.561261 17652 ts_tablet_manager.cc:1398] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20240502 01:55:25.562031 17652 tablet_bootstrap.cc:492] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e: Bootstrap starting.
I20240502 01:55:25.563655 17652 tablet_bootstrap.cc:654] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:25.565418 17657 tablet_bootstrap.cc:492] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367: No bootstrap required, opened a new log
I20240502 01:55:25.565613 17657 ts_tablet_manager.cc:1367] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20240502 01:55:25.566383 17657 raft_consensus.cc:348] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.566686 17657 raft_consensus.cc:374] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:25.566763 17657 raft_consensus.cc:729] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: FOLLOWER
I20240502 01:55:25.567003 17657 consensus_queue.cc:260] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.567232 17653 ts_tablet_manager.cc:1398] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a: Time spent starting tablet: real 0.012s user 0.000s sys 0.010s
I20240502 01:55:25.567709 17659 tablet_bootstrap.cc:492] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304: Bootstrap starting.
I20240502 01:55:25.567874 17651 raft_consensus.cc:348] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.568181 17651 raft_consensus.cc:374] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:25.568262 17651 raft_consensus.cc:729] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fcb8d12cb4ab28ec821c9473adeeb, State: Initialized, Role: FOLLOWER
I20240502 01:55:25.568574 17651 consensus_queue.cc:260] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.569772 17651 ts_tablet_manager.cc:1398] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20240502 01:55:25.561131 17654 raft_consensus.cc:348] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:25.570199 17654 raft_consensus.cc:374] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:25.570317 17654 raft_consensus.cc:729] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Initialized, Role: FOLLOWER
I20240502 01:55:25.570518 17656 tablet_bootstrap.cc:492] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad: No bootstrap required, opened a new log
I20240502 01:55:25.570698 17656 ts_tablet_manager.cc:1367] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.042s user 0.010s sys 0.007s
I20240502 01:55:25.570652 17654 consensus_queue.cc:260] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:25.571399 17654 ts_tablet_manager.cc:1398] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 01:55:25.571481 17653 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a: Bootstrap starting.
I20240502 01:55:25.571441 17656 raft_consensus.cc:348] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:25.571697 17656 raft_consensus.cc:374] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:25.571789 17656 raft_consensus.cc:729] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: FOLLOWER
I20240502 01:55:25.572036 17656 consensus_queue.cc:260] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:25.572240 17654 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9: Bootstrap starting.
I20240502 01:55:25.572618 17656 ts_tablet_manager.cc:1398] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:55:25.572997 17651 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb: Bootstrap starting.
I20240502 01:55:25.573539 17657 ts_tablet_manager.cc:1398] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 01:55:25.574811 17659 tablet_bootstrap.cc:654] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:25.574977 17653 tablet_bootstrap.cc:654] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:25.575141 17651 tablet_bootstrap.cc:654] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:25.576282 17657 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:25.577394 17654 tablet_bootstrap.cc:654] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:25.577831 17657 tablet_bootstrap.cc:654] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:25.578951 17656 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:25.579556 17652 tablet_bootstrap.cc:492] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e: No bootstrap required, opened a new log
I20240502 01:55:25.579797 17652 ts_tablet_manager.cc:1367] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240502 01:55:25.579855 17659 tablet_bootstrap.cc:492] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304: No bootstrap required, opened a new log
I20240502 01:55:25.579999 17665 raft_consensus.cc:482] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:25.580077 17659 ts_tablet_manager.cc:1367] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 01:55:25.580408 17656 tablet_bootstrap.cc:654] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:25.580157 17665 raft_consensus.cc:504] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:25.580610 17652 raft_consensus.cc:348] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:25.580879 17652 raft_consensus.cc:374] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:25.580951 17652 raft_consensus.cc:729] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4d50250184223992325c86cb6c68e, State: Initialized, Role: FOLLOWER
I20240502 01:55:25.580837 17659 raft_consensus.cc:348] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.581130 17659 raft_consensus.cc:374] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:25.581228 17659 raft_consensus.cc:729] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Initialized, Role: FOLLOWER
I20240502 01:55:25.581284 17652 consensus_queue.cc:260] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:25.581496 17659 consensus_queue.cc:260] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.582075 17659 ts_tablet_manager.cc:1398] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:55:25.583091 17651 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb: No bootstrap required, opened a new log
I20240502 01:55:25.583237 17651 ts_tablet_manager.cc:1367] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20240502 01:55:25.583554 17665 leader_election.cc:288] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425)
I20240502 01:55:25.584235 17651 raft_consensus.cc:348] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.584616 17651 raft_consensus.cc:374] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:25.584718 17651 raft_consensus.cc:729] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fcb8d12cb4ab28ec821c9473adeeb, State: Initialized, Role: FOLLOWER
I20240502 01:55:25.585003 17651 consensus_queue.cc:260] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.585767 17651 ts_tablet_manager.cc:1398] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:25.587766 17653 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a: No bootstrap required, opened a new log
I20240502 01:55:25.587971 17653 ts_tablet_manager.cc:1367] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20240502 01:55:25.588821 17653 raft_consensus.cc:348] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.589167 17653 raft_consensus.cc:374] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:25.589254 17653 raft_consensus.cc:729] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Initialized, Role: FOLLOWER
I20240502 01:55:25.589538 17653 consensus_queue.cc:260] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.590160 17653 ts_tablet_manager.cc:1398] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:25.591202 17653 tablet_bootstrap.cc:492] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a: Bootstrap starting.
I20240502 01:55:25.592770 17653 tablet_bootstrap.cc:654] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:25.595276 17652 ts_tablet_manager.cc:1398] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240502 01:55:25.598969 17652 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e: Bootstrap starting.
I20240502 01:55:25.604578 17652 tablet_bootstrap.cc:654] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:25.607869 17656 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad: No bootstrap required, opened a new log
I20240502 01:55:25.608150 17656 ts_tablet_manager.cc:1367] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.001s
I20240502 01:55:25.608948 17656 raft_consensus.cc:348] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.611511 17656 raft_consensus.cc:374] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:25.611928 17656 raft_consensus.cc:729] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: FOLLOWER
I20240502 01:55:25.613205 17656 consensus_queue.cc:260] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.616273 17652 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e: No bootstrap required, opened a new log
I20240502 01:55:25.616861 17656 ts_tablet_manager.cc:1398] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:55:25.617846 17656 tablet_bootstrap.cc:492] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:25.618091 17652 ts_tablet_manager.cc:1367] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.003s
I20240502 01:55:25.618193 17653 tablet_bootstrap.cc:492] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a: No bootstrap required, opened a new log
I20240502 01:55:25.618433 17653 ts_tablet_manager.cc:1367] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.021s
I20240502 01:55:25.619323 17653 raft_consensus.cc:348] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:25.619693 17653 raft_consensus.cc:374] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:25.619854 17653 raft_consensus.cc:729] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Initialized, Role: FOLLOWER
I20240502 01:55:25.620234 17653 consensus_queue.cc:260] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:25.621069 17652 raft_consensus.cc:348] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.622602 17656 tablet_bootstrap.cc:654] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:25.622750 17652 raft_consensus.cc:374] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:25.623041 17652 raft_consensus.cc:729] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4d50250184223992325c86cb6c68e, State: Initialized, Role: FOLLOWER
I20240502 01:55:25.623457 17652 consensus_queue.cc:260] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.626904 17652 ts_tablet_manager.cc:1398] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:55:25.631007 17656 tablet_bootstrap.cc:492] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad: No bootstrap required, opened a new log
I20240502 01:55:25.631682 17656 ts_tablet_manager.cc:1367] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20240502 01:55:25.632696 17656 raft_consensus.cc:348] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.634054 17656 raft_consensus.cc:374] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:25.634301 17656 raft_consensus.cc:729] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: FOLLOWER
I20240502 01:55:25.634923 17656 consensus_queue.cc:260] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.636180 17656 ts_tablet_manager.cc:1398] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 01:55:25.660636 17650 raft_consensus.cc:482] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:25.660838 17650 raft_consensus.cc:504] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.662124 17650 leader_election.cc:288] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:25.663357 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512eab02225d4544a342811f0a7359aa" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:25.663633 16225 raft_consensus.cc:2454] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:25.664758 16620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512eab02225d4544a342811f0a7359aa" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
W20240502 01:55:25.665791 16690 leader_election.cc:341] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 703f6f043c6041a4b629933690580367 (127.31.161.196:40425): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:25.666702 16358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512eab02225d4544a342811f0a7359aa" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:25.666954 16358 raft_consensus.cc:2454] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:25.667927 16884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512eab02225d4544a342811f0a7359aa" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:25.668368 16884 raft_consensus.cc:2454] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:25.669157 16690 leader_election.cc:302] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 6b9f9aaa1c86420190af75270914449a, cdd4d50250184223992325c86cb6c68e, dc9fcb8d12cb4ab28ec821c9473adeeb, ed2e7e4188d0400b8689e28bbf5096ad; no voters: 703f6f043c6041a4b629933690580367
I20240502 01:55:25.669548 17650 raft_consensus.cc:2781] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:25.669657 17650 raft_consensus.cc:482] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:25.669734 17650 raft_consensus.cc:3037] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:25.672700 17650 raft_consensus.cc:504] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.674448 16620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512eab02225d4544a342811f0a7359aa" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367"
W20240502 01:55:25.675148 16690 leader_election.cc:341] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 703f6f043c6041a4b629933690580367 (127.31.161.196:40425): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:25.676074 16884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512eab02225d4544a342811f0a7359aa" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a"
I20240502 01:55:25.676293 16884 raft_consensus.cc:3037] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:25.680672 16884 raft_consensus.cc:2454] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:25.680691 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512eab02225d4544a342811f0a7359aa" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
W20240502 01:55:25.681573 16687 leader_election.cc:341] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:25.682163 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512eab02225d4544a342811f0a7359aa" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e"
I20240502 01:55:25.682266 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512eab02225d4544a342811f0a7359aa" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9"
I20240502 01:55:25.682415 16225 raft_consensus.cc:3037] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:55:25.682732 16687 leader_election.cc:341] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:25.686352 16225 raft_consensus.cc:2454] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:25.690093 16358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512eab02225d4544a342811f0a7359aa" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
I20240502 01:55:25.690338 16358 raft_consensus.cc:3037] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:25.691597 16491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512eab02225d4544a342811f0a7359aa" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:25.691852 16491 raft_consensus.cc:2454] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:25.693727 16358 raft_consensus.cc:2454] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:25.694319 16690 leader_election.cc:302] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 6b9f9aaa1c86420190af75270914449a, cdd4d50250184223992325c86cb6c68e, dc9fcb8d12cb4ab28ec821c9473adeeb, ed2e7e4188d0400b8689e28bbf5096ad; no voters: 703f6f043c6041a4b629933690580367, a6575ebf978a4631a13ba5c243c3c4f9
I20240502 01:55:25.694658 17615 raft_consensus.cc:2781] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:25.695013 17650 leader_election.cc:288] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 election: Requested vote from peers 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:25.713845 17615 raft_consensus.cc:686] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Becoming Leader. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Running, Role: LEADER
I20240502 01:55:25.714285 17615 consensus_queue.cc:237] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.720443 17672 raft_consensus.cc:482] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:25.720633 17672 raft_consensus.cc:504] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.721028 17657 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367: No bootstrap required, opened a new log
I20240502 01:55:25.721226 17657 ts_tablet_manager.cc:1367] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.145s user 0.031s sys 0.078s
I20240502 01:55:25.722044 17657 raft_consensus.cc:348] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.722357 17657 raft_consensus.cc:374] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:25.722443 17657 raft_consensus.cc:729] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: FOLLOWER
I20240502 01:55:25.722690 17657 consensus_queue.cc:260] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.723433 17672 leader_election.cc:288] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:25.723958 17657 ts_tablet_manager.cc:1398] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:55:25.727136 17664 raft_consensus.cc:482] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:25.727315 17664 raft_consensus.cc:504] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.729792 17664 leader_election.cc:288] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:25.730846 16358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297561b66dd7472ab817c2a23723565c" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:25.731103 16358 raft_consensus.cc:2454] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:25.732872 16884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297561b66dd7472ab817c2a23723565c" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:25.733129 16884 raft_consensus.cc:2454] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:25.735029 16491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297561b66dd7472ab817c2a23723565c" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:25.735285 16491 raft_consensus.cc:2454] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:25.739621 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297561b66dd7472ab817c2a23723565c" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:25.739858 16754 raft_consensus.cc:2454] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:25.742779 16491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512eab02225d4544a342811f0a7359aa" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304"
I20240502 01:55:25.742998 16491 raft_consensus.cc:3037] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:25.745853 16491 raft_consensus.cc:2454] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:25.748215 17654 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9: No bootstrap required, opened a new log
I20240502 01:55:25.748406 17654 ts_tablet_manager.cc:1367] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9: Time spent bootstrapping tablet: real 0.176s user 0.025s sys 0.021s
I20240502 01:55:25.749368 17654 raft_consensus.cc:348] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.749651 17654 raft_consensus.cc:374] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:25.749730 17654 raft_consensus.cc:729] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Initialized, Role: FOLLOWER
I20240502 01:55:25.749964 17654 consensus_queue.cc:260] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.751454 16950 leader_election.cc:302] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 6b9f9aaa1c86420190af75270914449a, a6575ebf978a4631a13ba5c243c3c4f9, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:25.751894 17665 raft_consensus.cc:2781] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:25.752007 17665 raft_consensus.cc:482] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:25.752094 17665 raft_consensus.cc:3037] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:25.755059 17665 raft_consensus.cc:504] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:25.756264 17665 leader_election.cc:288] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [CANDIDATE]: Term 1 election: Requested vote from peers dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425)
I20240502 01:55:25.757197 16491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297561b66dd7472ab817c2a23723565c" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304"
I20240502 01:55:25.757416 16491 raft_consensus.cc:3037] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:25.757452 16884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297561b66dd7472ab817c2a23723565c" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a"
I20240502 01:55:25.757692 16884 raft_consensus.cc:3037] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:25.760251 16491 raft_consensus.cc:2454] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:25.760466 16884 raft_consensus.cc:2454] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:25.761440 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297561b66dd7472ab817c2a23723565c" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad"
I20240502 01:55:25.761677 16754 raft_consensus.cc:3037] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:25.764484 16754 raft_consensus.cc:2454] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:25.765297 16358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297561b66dd7472ab817c2a23723565c" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
I20240502 01:55:25.765453 17654 ts_tablet_manager.cc:1398] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20240502 01:55:25.765555 16358 raft_consensus.cc:3037] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:25.766752 17653 ts_tablet_manager.cc:1398] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a: Time spent starting tablet: real 0.148s user 0.020s sys 0.065s
I20240502 01:55:25.768949 16358 raft_consensus.cc:2454] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:25.769902 16952 leader_election.cc:302] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, a6575ebf978a4631a13ba5c243c3c4f9, dc9fcb8d12cb4ab28ec821c9473adeeb, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:25.770211 17665 raft_consensus.cc:2781] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:25.779424 17692 raft_consensus.cc:482] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:25.779640 17692 raft_consensus.cc:504] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.791006 17692 leader_election.cc:288] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:25.797253 16080 catalog_manager.cc:5484] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: term changed from 0 to 1, leader changed from <none> to ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197). New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:25.809872 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:25.810128 16225 raft_consensus.cc:2454] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:25.814275 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297561b66dd7472ab817c2a23723565c" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:25.814500 16225 raft_consensus.cc:2454] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:25.815153 16358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "cdd4d50250184223992325c86cb6c68e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:25.816135 16358 raft_consensus.cc:2454] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdd4d50250184223992325c86cb6c68e in term 0.
I20240502 01:55:25.816516 17664 raft_consensus.cc:482] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:25.816763 17664 raft_consensus.cc:504] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:25.817955 17664 leader_election.cc:288] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:25.818707 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297561b66dd7472ab817c2a23723565c" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:25.818923 16225 raft_consensus.cc:2454] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:25.820302 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:25.820588 16754 raft_consensus.cc:2454] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:25.820529 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297561b66dd7472ab817c2a23723565c" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:25.820881 16753 raft_consensus.cc:2379] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 703f6f043c6041a4b629933690580367 in current term 1: Already voted for candidate a6575ebf978a4631a13ba5c243c3c4f9 in this term.
I20240502 01:55:25.821761 16491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:25.821997 16491 raft_consensus.cc:2454] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:25.822302 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297561b66dd7472ab817c2a23723565c" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:25.822564 16490 raft_consensus.cc:2379] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 703f6f043c6041a4b629933690580367 in current term 1: Already voted for candidate a6575ebf978a4631a13ba5c243c3c4f9 in this term.
I20240502 01:55:25.825407 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "cdd4d50250184223992325c86cb6c68e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:25.825610 16490 raft_consensus.cc:2454] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdd4d50250184223992325c86cb6c68e in term 0.
I20240502 01:55:25.828785 17665 raft_consensus.cc:686] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Becoming Leader. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Running, Role: LEADER
I20240502 01:55:25.829142 16555 leader_election.cc:302] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 703f6f043c6041a4b629933690580367, cdd4d50250184223992325c86cb6c68e, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:25.829452 17665 consensus_queue.cc:237] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:25.830927 17664 raft_consensus.cc:2781] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:25.831040 17664 raft_consensus.cc:482] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:25.831144 17664 raft_consensus.cc:3037] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:25.834110 17664 raft_consensus.cc:504] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.835677 17664 leader_election.cc:288] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 election: Requested vote from peers ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:25.835907 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad"
I20240502 01:55:25.836112 16753 raft_consensus.cc:3037] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:25.836273 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e"
I20240502 01:55:25.836484 16225 raft_consensus.cc:3037] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:25.838392 16884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:25.838632 16884 raft_consensus.cc:2454] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:25.839004 16753 raft_consensus.cc:2454] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
I20240502 01:55:25.839226 16225 raft_consensus.cc:2454] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
I20240502 01:55:25.839507 16884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297561b66dd7472ab817c2a23723565c" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:25.839730 16884 raft_consensus.cc:2379] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 703f6f043c6041a4b629933690580367 in current term 1: Already voted for candidate a6575ebf978a4631a13ba5c243c3c4f9 in this term.
I20240502 01:55:25.840193 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304"
I20240502 01:55:25.840427 16490 raft_consensus.cc:3037] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:25.840528 16884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a"
I20240502 01:55:25.840679 16884 raft_consensus.cc:3037] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:25.842698 16490 raft_consensus.cc:2454] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
I20240502 01:55:25.843082 16555 leader_election.cc:302] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 703f6f043c6041a4b629933690580367, cdd4d50250184223992325c86cb6c68e, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:25.843360 16884 raft_consensus.cc:2454] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
I20240502 01:55:25.843394 17664 raft_consensus.cc:2781] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:25.843722 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "cdd4d50250184223992325c86cb6c68e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:25.843747 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297561b66dd7472ab817c2a23723565c" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e"
I20240502 01:55:25.843986 16753 raft_consensus.cc:2379] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cdd4d50250184223992325c86cb6c68e in current term 1: Already voted for candidate 703f6f043c6041a4b629933690580367 in this term.
I20240502 01:55:25.843955 16225 raft_consensus.cc:3037] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:25.843972 16884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "cdd4d50250184223992325c86cb6c68e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:25.844247 16884 raft_consensus.cc:2379] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cdd4d50250184223992325c86cb6c68e in current term 1: Already voted for candidate 703f6f043c6041a4b629933690580367 in this term.
I20240502 01:55:25.845422 16620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:25.846599 16225 raft_consensus.cc:2454] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:25.847153 16358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:25.847393 16358 raft_consensus.cc:2454] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 0.
I20240502 01:55:25.847555 16622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297561b66dd7472ab817c2a23723565c" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:25.847787 16622 raft_consensus.cc:2454] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:25.848085 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297561b66dd7472ab817c2a23723565c" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367"
I20240502 01:55:25.848258 16621 raft_consensus.cc:3037] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:25.848229 16358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:25.848441 16358 raft_consensus.cc:2454] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:25.848453 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297561b66dd7472ab817c2a23723565c" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:25.848733 16359 raft_consensus.cc:2379] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 703f6f043c6041a4b629933690580367 in current term 1: Already voted for candidate a6575ebf978a4631a13ba5c243c3c4f9 in this term.
I20240502 01:55:25.849418 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
I20240502 01:55:25.849572 16359 raft_consensus.cc:3037] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:25.851011 16621 raft_consensus.cc:2454] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:25.852159 16359 raft_consensus.cc:2454] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
I20240502 01:55:25.853515 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "cdd4d50250184223992325c86cb6c68e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:25.854583 16558 leader_election.cc:302] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 703f6f043c6041a4b629933690580367, cdd4d50250184223992325c86cb6c68e; no voters: 39059854b64045698ded1ba4e8d47304, 6b9f9aaa1c86420190af75270914449a, dc9fcb8d12cb4ab28ec821c9473adeeb, ed2e7e4188d0400b8689e28bbf5096ad
I20240502 01:55:25.858392 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "cdd4d50250184223992325c86cb6c68e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:25.858647 17016 raft_consensus.cc:2454] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdd4d50250184223992325c86cb6c68e in term 0.
I20240502 01:55:25.859287 17726 raft_consensus.cc:2726] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:55:25.859167 16158 leader_election.cc:302] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, a6575ebf978a4631a13ba5c243c3c4f9, cdd4d50250184223992325c86cb6c68e, dc9fcb8d12cb4ab28ec821c9473adeeb; no voters: 6b9f9aaa1c86420190af75270914449a, ed2e7e4188d0400b8689e28bbf5096ad
I20240502 01:55:25.859412 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:25.859623 17016 raft_consensus.cc:2454] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:25.859680 17672 raft_consensus.cc:2741] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240502 01:55:25.860545 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "297561b66dd7472ab817c2a23723565c" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:25.861126 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9"
I20240502 01:55:25.861299 17016 raft_consensus.cc:3037] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:25.861564 17664 raft_consensus.cc:686] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Becoming Leader. State: Replica: 703f6f043c6041a4b629933690580367, State: Running, Role: LEADER
I20240502 01:55:25.862613 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:25.862907 16753 raft_consensus.cc:2379] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b9f9aaa1c86420190af75270914449a in current term 1: Already voted for candidate 703f6f043c6041a4b629933690580367 in this term.
I20240502 01:55:25.862612 16080 catalog_manager.cc:5484] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199). New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:25.863827 17016 raft_consensus.cc:2454] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
I20240502 01:55:25.864125 17015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:25.864452 17015 raft_consensus.cc:2379] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b9f9aaa1c86420190af75270914449a in current term 1: Already voted for candidate 703f6f043c6041a4b629933690580367 in this term.
I20240502 01:55:25.865631 17664 consensus_queue.cc:237] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:25.870918 16080 catalog_manager.cc:5484] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 reported cstate change: term changed from 0 to 1, leader changed from <none> to 703f6f043c6041a4b629933690580367 (127.31.161.196). New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:25.872819 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:25.873135 16490 raft_consensus.cc:2379] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b9f9aaa1c86420190af75270914449a in current term 1: Already voted for candidate 703f6f043c6041a4b629933690580367 in this term.
I20240502 01:55:25.873330 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f53b466fba6a4655a63a559b0ea56f77" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:25.873594 16225 raft_consensus.cc:2379] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b9f9aaa1c86420190af75270914449a in current term 1: Already voted for candidate 703f6f043c6041a4b629933690580367 in this term.
I20240502 01:55:25.873765 16818 leader_election.cc:302] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 6b9f9aaa1c86420190af75270914449a, dc9fcb8d12cb4ab28ec821c9473adeeb; no voters: 39059854b64045698ded1ba4e8d47304, 703f6f043c6041a4b629933690580367, a6575ebf978a4631a13ba5c243c3c4f9, ed2e7e4188d0400b8689e28bbf5096ad
I20240502 01:55:25.874271 17692 raft_consensus.cc:2726] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:55:26.095341 17615 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:26.133670 17664 raft_consensus.cc:482] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:26.133883 17664 raft_consensus.cc:504] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:26.135347 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512eab02225d4544a342811f0a7359aa" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:26.135686 17015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512eab02225d4544a342811f0a7359aa" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:26.135949 17015 raft_consensus.cc:2454] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:26.136107 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512eab02225d4544a342811f0a7359aa" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:26.136292 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512eab02225d4544a342811f0a7359aa" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:26.136561 16359 raft_consensus.cc:2379] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 703f6f043c6041a4b629933690580367 in current term 1: Already voted for candidate ed2e7e4188d0400b8689e28bbf5096ad in this term.
I20240502 01:55:26.136636 17664 leader_election.cc:288] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:26.137136 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512eab02225d4544a342811f0a7359aa" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:26.137248 16884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "512eab02225d4544a342811f0a7359aa" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:26.137423 16225 raft_consensus.cc:2379] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 703f6f043c6041a4b629933690580367 in current term 1: Already voted for candidate ed2e7e4188d0400b8689e28bbf5096ad in this term.
I20240502 01:55:26.137545 16884 raft_consensus.cc:2379] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 703f6f043c6041a4b629933690580367 in current term 1: Already voted for candidate ed2e7e4188d0400b8689e28bbf5096ad in this term.
I20240502 01:55:26.137899 16557 leader_election.cc:302] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 703f6f043c6041a4b629933690580367, a6575ebf978a4631a13ba5c243c3c4f9; no voters: 39059854b64045698ded1ba4e8d47304, cdd4d50250184223992325c86cb6c68e, dc9fcb8d12cb4ab28ec821c9473adeeb, ed2e7e4188d0400b8689e28bbf5096ad
I20240502 01:55:26.138204 17664 raft_consensus.cc:3037] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:26.140635 17664 raft_consensus.cc:2726] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:55:26.141562 17615 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:26.157816 17615 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:26.185585 17615 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:26.191473 17015 raft_consensus.cc:3037] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:26.191795 17650 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:26.196338 17698 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:26.254894 17665 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:26.255312 17714 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:26.256254 17664 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:26.326278 17729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:26.326575 17729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:55:26.328719 17664 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:26.335285 17729 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 01:55:26.387125 17714 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:26.388259 17714 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:26.393733 17714 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:26.400245 17665 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:26.402272 17664 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:26.463603 17664 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:26.472558 17664 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:26.481999 17779 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 445.98 ms
time per row: 0.22299 ms
I20240502 01:55:26.932965 16075 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58366:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 01:55:26.933758 16075 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240502 01:55:26.943571 16601 tablet_service.cc:1467] Processing CreateTablet for tablet bab1fc6db80c4e53a4f0cacccca74096 (DEFAULT_TABLE table=rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:26.943828 16733 tablet_service.cc:1467] Processing CreateTablet for tablet bab1fc6db80c4e53a4f0cacccca74096 (DEFAULT_TABLE table=rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:26.944085 16601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab1fc6db80c4e53a4f0cacccca74096. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:26.944284 16733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab1fc6db80c4e53a4f0cacccca74096. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:26.944697 16863 tablet_service.cc:1467] Processing CreateTablet for tablet bab1fc6db80c4e53a4f0cacccca74096 (DEFAULT_TABLE table=rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:26.945068 16996 tablet_service.cc:1467] Processing CreateTablet for tablet bab1fc6db80c4e53a4f0cacccca74096 (DEFAULT_TABLE table=rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:26.945509 16996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab1fc6db80c4e53a4f0cacccca74096. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:26.945554 16863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab1fc6db80c4e53a4f0cacccca74096. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:26.945556 16205 tablet_service.cc:1467] Processing CreateTablet for tablet bab1fc6db80c4e53a4f0cacccca74096 (DEFAULT_TABLE table=rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:26.946013 16205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab1fc6db80c4e53a4f0cacccca74096. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:26.952766 16338 tablet_service.cc:1467] Processing CreateTablet for tablet bab1fc6db80c4e53a4f0cacccca74096 (DEFAULT_TABLE table=rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:26.953282 16338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab1fc6db80c4e53a4f0cacccca74096. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:26.953917 16204 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb6f3c696db4acf8a048f8ddc9e1230 (DEFAULT_TABLE table=rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:26.954366 16204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb6f3c696db4acf8a048f8ddc9e1230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:26.957357 16865 tablet_service.cc:1467] Processing CreateTablet for tablet 18e676bcd8b04f18a6814d6ebbd1b488 (DEFAULT_TABLE table=rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:26.957470 16864 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb6f3c696db4acf8a048f8ddc9e1230 (DEFAULT_TABLE table=rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:26.957799 16865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18e676bcd8b04f18a6814d6ebbd1b488. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:26.959164 16600 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb6f3c696db4acf8a048f8ddc9e1230 (DEFAULT_TABLE table=rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:26.959627 16600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb6f3c696db4acf8a048f8ddc9e1230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:26.961448 16602 tablet_service.cc:1467] Processing CreateTablet for tablet 18e676bcd8b04f18a6814d6ebbd1b488 (DEFAULT_TABLE table=rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:26.961810 16602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18e676bcd8b04f18a6814d6ebbd1b488. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:26.964321 16470 tablet_service.cc:1467] Processing CreateTablet for tablet bab1fc6db80c4e53a4f0cacccca74096 (DEFAULT_TABLE table=rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:26.964746 16470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab1fc6db80c4e53a4f0cacccca74096. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:26.964813 16994 tablet_service.cc:1467] Processing CreateTablet for tablet 18e676bcd8b04f18a6814d6ebbd1b488 (DEFAULT_TABLE table=rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:26.965251 16994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18e676bcd8b04f18a6814d6ebbd1b488. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:26.965718 16864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb6f3c696db4acf8a048f8ddc9e1230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:26.966041 16471 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb6f3c696db4acf8a048f8ddc9e1230 (DEFAULT_TABLE table=rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:26.966408 16471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb6f3c696db4acf8a048f8ddc9e1230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:26.969411 16732 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb6f3c696db4acf8a048f8ddc9e1230 (DEFAULT_TABLE table=rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:26.969854 16732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb6f3c696db4acf8a048f8ddc9e1230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:26.971586 17834 tablet_bootstrap.cc:492] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9: Bootstrap starting.
I20240502 01:55:26.973462 16734 tablet_service.cc:1467] Processing CreateTablet for tablet 18e676bcd8b04f18a6814d6ebbd1b488 (DEFAULT_TABLE table=rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:26.973614 16203 tablet_service.cc:1467] Processing CreateTablet for tablet 18e676bcd8b04f18a6814d6ebbd1b488 (DEFAULT_TABLE table=rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:26.973840 16734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18e676bcd8b04f18a6814d6ebbd1b488. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:26.974018 16203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18e676bcd8b04f18a6814d6ebbd1b488. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:26.974435 16469 tablet_service.cc:1467] Processing CreateTablet for tablet 18e676bcd8b04f18a6814d6ebbd1b488 (DEFAULT_TABLE table=rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:26.974826 16469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18e676bcd8b04f18a6814d6ebbd1b488. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:26.974895 16995 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb6f3c696db4acf8a048f8ddc9e1230 (DEFAULT_TABLE table=rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:26.975319 16995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb6f3c696db4acf8a048f8ddc9e1230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:26.978286 16337 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb6f3c696db4acf8a048f8ddc9e1230 (DEFAULT_TABLE table=rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:26.978387 16339 tablet_service.cc:1467] Processing CreateTablet for tablet 18e676bcd8b04f18a6814d6ebbd1b488 (DEFAULT_TABLE table=rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:26.978730 16337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb6f3c696db4acf8a048f8ddc9e1230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:26.979923 16339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18e676bcd8b04f18a6814d6ebbd1b488. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:26.982496 17833 tablet_bootstrap.cc:492] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:26.984174 17833 tablet_bootstrap.cc:654] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:26.994446 17836 tablet_bootstrap.cc:492] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:26.996894 17836 tablet_bootstrap.cc:654] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:26.997924 17834 tablet_bootstrap.cc:654] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:27.005216 17842 tablet_bootstrap.cc:492] T 9eb6f3c696db4acf8a048f8ddc9e1230 P dc9fcb8d12cb4ab28ec821c9473adeeb: Bootstrap starting.
I20240502 01:55:27.006922 17842 tablet_bootstrap.cc:654] T 9eb6f3c696db4acf8a048f8ddc9e1230 P dc9fcb8d12cb4ab28ec821c9473adeeb: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:27.010258 17836 tablet_bootstrap.cc:492] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367: No bootstrap required, opened a new log
I20240502 01:55:27.010427 17836 ts_tablet_manager.cc:1367] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.003s
I20240502 01:55:27.010584 17835 tablet_bootstrap.cc:492] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e: Bootstrap starting.
I20240502 01:55:27.011246 17836 raft_consensus.cc:348] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.011543 17836 raft_consensus.cc:374] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:27.011631 17836 raft_consensus.cc:729] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: FOLLOWER
I20240502 01:55:27.012205 17835 tablet_bootstrap.cc:654] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:27.012483 17834 tablet_bootstrap.cc:492] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9: No bootstrap required, opened a new log
I20240502 01:55:27.012650 17834 ts_tablet_manager.cc:1367] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent bootstrapping tablet: real 0.041s user 0.000s sys 0.004s
I20240502 01:55:27.013593 17834 raft_consensus.cc:348] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.013916 17834 raft_consensus.cc:374] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:27.014003 17834 raft_consensus.cc:729] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Initialized, Role: FOLLOWER
I20240502 01:55:27.014065 17842 tablet_bootstrap.cc:492] T 9eb6f3c696db4acf8a048f8ddc9e1230 P dc9fcb8d12cb4ab28ec821c9473adeeb: No bootstrap required, opened a new log
I20240502 01:55:27.014219 17842 ts_tablet_manager.cc:1367] T 9eb6f3c696db4acf8a048f8ddc9e1230 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240502 01:55:27.014891 17834 consensus_queue.cc:260] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.014953 17842 raft_consensus.cc:348] T 9eb6f3c696db4acf8a048f8ddc9e1230 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:27.015210 17842 raft_consensus.cc:374] T 9eb6f3c696db4acf8a048f8ddc9e1230 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:27.015301 17842 raft_consensus.cc:729] T 9eb6f3c696db4acf8a048f8ddc9e1230 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fcb8d12cb4ab28ec821c9473adeeb, State: Initialized, Role: FOLLOWER
I20240502 01:55:27.015369 17836 consensus_queue.cc:260] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.016080 17836 ts_tablet_manager.cc:1398] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 01:55:27.016090 17833 tablet_bootstrap.cc:492] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad: No bootstrap required, opened a new log
I20240502 01:55:27.017359 17834 ts_tablet_manager.cc:1398] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:55:27.016248 17833 ts_tablet_manager.cc:1367] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.011s
I20240502 01:55:27.017680 17834 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9: Bootstrap starting.
I20240502 01:55:27.018314 17833 raft_consensus.cc:348] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.018636 17833 raft_consensus.cc:374] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:27.018728 17833 raft_consensus.cc:729] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: FOLLOWER
I20240502 01:55:27.019649 17834 tablet_bootstrap.cc:654] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:27.020013 17842 consensus_queue.cc:260] T 9eb6f3c696db4acf8a048f8ddc9e1230 P dc9fcb8d12cb4ab28ec821c9473adeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:27.020242 17836 tablet_bootstrap.cc:492] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:27.020851 17833 consensus_queue.cc:260] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.023012 17833 ts_tablet_manager.cc:1398] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:55:27.023310 17833 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:27.023923 17841 tablet_bootstrap.cc:492] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 39059854b64045698ded1ba4e8d47304: Bootstrap starting.
I20240502 01:55:27.024825 17836 tablet_bootstrap.cc:654] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 703f6f043c6041a4b629933690580367: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:27.024935 17835 tablet_bootstrap.cc:492] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e: No bootstrap required, opened a new log
I20240502 01:55:27.025184 17835 ts_tablet_manager.cc:1367] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20240502 01:55:27.026487 17835 raft_consensus.cc:348] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.026774 17835 raft_consensus.cc:374] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:27.027313 17837 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a: Bootstrap starting.
I20240502 01:55:27.027372 17833 tablet_bootstrap.cc:654] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:27.028184 17841 tablet_bootstrap.cc:654] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 39059854b64045698ded1ba4e8d47304: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:27.028630 17835 raft_consensus.cc:729] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4d50250184223992325c86cb6c68e, State: Initialized, Role: FOLLOWER
I20240502 01:55:27.028920 17835 consensus_queue.cc:260] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.029834 17837 tablet_bootstrap.cc:654] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:27.029844 17833 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad: No bootstrap required, opened a new log
I20240502 01:55:27.029999 17833 ts_tablet_manager.cc:1367] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240502 01:55:27.030676 17833 raft_consensus.cc:348] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.030975 17833 raft_consensus.cc:374] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:27.031059 17833 raft_consensus.cc:729] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: FOLLOWER
I20240502 01:55:27.031149 17835 ts_tablet_manager.cc:1398] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:55:27.031498 17835 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e: Bootstrap starting.
I20240502 01:55:27.032003 17836 tablet_bootstrap.cc:492] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 703f6f043c6041a4b629933690580367: No bootstrap required, opened a new log
I20240502 01:55:27.032178 17836 ts_tablet_manager.cc:1367] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240502 01:55:27.032932 17836 raft_consensus.cc:348] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:27.033257 17836 raft_consensus.cc:374] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:27.033337 17836 raft_consensus.cc:729] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: FOLLOWER
I20240502 01:55:27.033591 17836 consensus_queue.cc:260] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:27.034204 17836 ts_tablet_manager.cc:1398] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:27.034811 17836 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:27.035244 17833 consensus_queue.cc:260] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.036834 17833 ts_tablet_manager.cc:1398] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:55:27.037904 17842 ts_tablet_manager.cc:1398] T 9eb6f3c696db4acf8a048f8ddc9e1230 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent starting tablet: real 0.024s user 0.000s sys 0.006s
I20240502 01:55:27.038197 17833 tablet_bootstrap.cc:492] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:27.039460 17842 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb: Bootstrap starting.
I20240502 01:55:27.040287 17836 tablet_bootstrap.cc:654] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:27.040371 17834 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9: No bootstrap required, opened a new log
I20240502 01:55:27.040566 17834 ts_tablet_manager.cc:1367] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.003s
I20240502 01:55:27.040988 17842 tablet_bootstrap.cc:654] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:27.041400 17834 raft_consensus.cc:348] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.041718 17834 raft_consensus.cc:374] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:27.041806 17834 raft_consensus.cc:729] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Initialized, Role: FOLLOWER
I20240502 01:55:27.042342 17833 tablet_bootstrap.cc:654] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:27.042183 17834 consensus_queue.cc:260] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.042929 17834 ts_tablet_manager.cc:1398] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:27.043367 17834 tablet_bootstrap.cc:492] T 9eb6f3c696db4acf8a048f8ddc9e1230 P a6575ebf978a4631a13ba5c243c3c4f9: Bootstrap starting.
I20240502 01:55:27.044816 17834 tablet_bootstrap.cc:654] T 9eb6f3c696db4acf8a048f8ddc9e1230 P a6575ebf978a4631a13ba5c243c3c4f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:27.046413 17835 tablet_bootstrap.cc:654] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:27.048339 17833 tablet_bootstrap.cc:492] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad: No bootstrap required, opened a new log
I20240502 01:55:27.048480 17833 ts_tablet_manager.cc:1367] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 01:55:27.049181 17833 raft_consensus.cc:348] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:27.049371 17841 tablet_bootstrap.cc:492] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 39059854b64045698ded1ba4e8d47304: No bootstrap required, opened a new log
I20240502 01:55:27.049440 17833 raft_consensus.cc:374] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:27.049517 17841 ts_tablet_manager.cc:1367] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 39059854b64045698ded1ba4e8d47304: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240502 01:55:27.049530 17833 raft_consensus.cc:729] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: FOLLOWER
I20240502 01:55:27.050264 17841 raft_consensus.cc:348] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:27.050554 17841 raft_consensus.cc:374] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:27.050645 17841 raft_consensus.cc:729] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Initialized, Role: FOLLOWER
I20240502 01:55:27.050900 17841 consensus_queue.cc:260] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 39059854b64045698ded1ba4e8d47304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:27.050907 17833 consensus_queue.cc:260] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:27.052085 17841 ts_tablet_manager.cc:1398] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 39059854b64045698ded1ba4e8d47304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:27.052129 17833 ts_tablet_manager.cc:1398] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:55:27.054188 17842 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb: No bootstrap required, opened a new log
I20240502 01:55:27.053429 17841 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304: Bootstrap starting.
I20240502 01:55:27.054368 17842 ts_tablet_manager.cc:1367] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.001s
I20240502 01:55:27.055001 17842 raft_consensus.cc:348] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.055243 17842 raft_consensus.cc:374] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:27.055320 17842 raft_consensus.cc:729] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fcb8d12cb4ab28ec821c9473adeeb, State: Initialized, Role: FOLLOWER
I20240502 01:55:27.055565 17842 consensus_queue.cc:260] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.056174 17842 ts_tablet_manager.cc:1398] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:55:27.056468 17842 tablet_bootstrap.cc:492] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb: Bootstrap starting.
I20240502 01:55:27.057152 17834 tablet_bootstrap.cc:492] T 9eb6f3c696db4acf8a048f8ddc9e1230 P a6575ebf978a4631a13ba5c243c3c4f9: No bootstrap required, opened a new log
I20240502 01:55:27.057315 17834 ts_tablet_manager.cc:1367] T 9eb6f3c696db4acf8a048f8ddc9e1230 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.004s
I20240502 01:55:27.057761 17842 tablet_bootstrap.cc:654] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:27.058235 17834 raft_consensus.cc:348] T 9eb6f3c696db4acf8a048f8ddc9e1230 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:27.058550 17834 raft_consensus.cc:374] T 9eb6f3c696db4acf8a048f8ddc9e1230 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:27.058665 17834 raft_consensus.cc:729] T 9eb6f3c696db4acf8a048f8ddc9e1230 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Initialized, Role: FOLLOWER
I20240502 01:55:27.059232 17836 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367: No bootstrap required, opened a new log
I20240502 01:55:27.059386 17836 ts_tablet_manager.cc:1367] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.002s
I20240502 01:55:27.060091 17836 raft_consensus.cc:348] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.060293 17836 raft_consensus.cc:374] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:27.060395 17836 raft_consensus.cc:729] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: FOLLOWER
I20240502 01:55:27.060636 17836 consensus_queue.cc:260] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.061208 17836 ts_tablet_manager.cc:1398] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:27.062824 17837 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a: No bootstrap required, opened a new log
I20240502 01:55:27.062997 17837 ts_tablet_manager.cc:1367] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.005s
I20240502 01:55:27.063575 17841 tablet_bootstrap.cc:654] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:27.063700 17842 tablet_bootstrap.cc:492] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb: No bootstrap required, opened a new log
I20240502 01:55:27.063834 17842 ts_tablet_manager.cc:1367] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240502 01:55:27.063855 17837 raft_consensus.cc:348] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.064122 17837 raft_consensus.cc:374] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:27.064227 17837 raft_consensus.cc:729] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Initialized, Role: FOLLOWER
I20240502 01:55:27.064458 17837 consensus_queue.cc:260] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.064530 17842 raft_consensus.cc:348] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.064798 17842 raft_consensus.cc:374] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:27.064890 17842 raft_consensus.cc:729] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fcb8d12cb4ab28ec821c9473adeeb, State: Initialized, Role: FOLLOWER
I20240502 01:55:27.065143 17842 consensus_queue.cc:260] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.065131 17834 consensus_queue.cc:260] T 9eb6f3c696db4acf8a048f8ddc9e1230 P a6575ebf978a4631a13ba5c243c3c4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:27.065730 17842 ts_tablet_manager.cc:1398] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:27.066457 17835 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e: No bootstrap required, opened a new log
I20240502 01:55:27.066640 17835 ts_tablet_manager.cc:1367] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20240502 01:55:27.067438 17835 raft_consensus.cc:348] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.067736 17835 raft_consensus.cc:374] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:27.067823 17835 raft_consensus.cc:729] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4d50250184223992325c86cb6c68e, State: Initialized, Role: FOLLOWER
I20240502 01:55:27.068114 17835 consensus_queue.cc:260] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.069478 17834 ts_tablet_manager.cc:1398] T 9eb6f3c696db4acf8a048f8ddc9e1230 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 01:55:27.069597 17837 ts_tablet_manager.cc:1398] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20240502 01:55:27.071360 17835 ts_tablet_manager.cc:1398] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 01:55:27.074107 17841 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304: No bootstrap required, opened a new log
I20240502 01:55:27.074266 17835 tablet_bootstrap.cc:492] T 9eb6f3c696db4acf8a048f8ddc9e1230 P cdd4d50250184223992325c86cb6c68e: Bootstrap starting.
I20240502 01:55:27.074295 17841 ts_tablet_manager.cc:1367] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.005s
I20240502 01:55:27.075166 17841 raft_consensus.cc:348] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.075495 17841 raft_consensus.cc:374] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:27.075577 17841 raft_consensus.cc:729] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Initialized, Role: FOLLOWER
I20240502 01:55:27.075681 17837 tablet_bootstrap.cc:492] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a: Bootstrap starting.
I20240502 01:55:27.075865 17841 consensus_queue.cc:260] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.076115 17835 tablet_bootstrap.cc:654] T 9eb6f3c696db4acf8a048f8ddc9e1230 P cdd4d50250184223992325c86cb6c68e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:27.077239 17837 tablet_bootstrap.cc:654] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:27.079357 17841 ts_tablet_manager.cc:1398] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 01:55:27.081888 17841 tablet_bootstrap.cc:492] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304: Bootstrap starting.
I20240502 01:55:27.083441 17841 tablet_bootstrap.cc:654] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:27.084059 17835 tablet_bootstrap.cc:492] T 9eb6f3c696db4acf8a048f8ddc9e1230 P cdd4d50250184223992325c86cb6c68e: No bootstrap required, opened a new log
I20240502 01:55:27.084256 17835 ts_tablet_manager.cc:1367] T 9eb6f3c696db4acf8a048f8ddc9e1230 P cdd4d50250184223992325c86cb6c68e: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240502 01:55:27.085183 17835 raft_consensus.cc:348] T 9eb6f3c696db4acf8a048f8ddc9e1230 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:27.085556 17835 raft_consensus.cc:374] T 9eb6f3c696db4acf8a048f8ddc9e1230 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:27.085698 17835 raft_consensus.cc:729] T 9eb6f3c696db4acf8a048f8ddc9e1230 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4d50250184223992325c86cb6c68e, State: Initialized, Role: FOLLOWER
I20240502 01:55:27.086004 17835 consensus_queue.cc:260] T 9eb6f3c696db4acf8a048f8ddc9e1230 P cdd4d50250184223992325c86cb6c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:27.086577 17827 raft_consensus.cc:482] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:27.086771 17827 raft_consensus.cc:504] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:27.087481 17837 tablet_bootstrap.cc:492] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a: No bootstrap required, opened a new log
I20240502 01:55:27.087733 17837 ts_tablet_manager.cc:1367] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240502 01:55:27.088124 17827 leader_election.cc:288] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:27.088722 17837 raft_consensus.cc:348] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:27.089089 17835 ts_tablet_manager.cc:1398] T 9eb6f3c696db4acf8a048f8ddc9e1230 P cdd4d50250184223992325c86cb6c68e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:55:27.089066 17837 raft_consensus.cc:374] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:27.089239 17837 raft_consensus.cc:729] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Initialized, Role: FOLLOWER
I20240502 01:55:27.089525 17837 consensus_queue.cc:260] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:27.090107 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb6f3c696db4acf8a048f8ddc9e1230" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:27.090255 17837 ts_tablet_manager.cc:1398] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:27.090387 16359 raft_consensus.cc:2454] T 9eb6f3c696db4acf8a048f8ddc9e1230 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:27.090802 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb6f3c696db4acf8a048f8ddc9e1230" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:27.091037 16619 raft_consensus.cc:2454] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:27.091347 17837 tablet_bootstrap.cc:492] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a: Bootstrap starting.
I20240502 01:55:27.091473 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb6f3c696db4acf8a048f8ddc9e1230" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:27.091701 16490 raft_consensus.cc:2454] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:27.092141 17015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb6f3c696db4acf8a048f8ddc9e1230" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:27.092391 17015 raft_consensus.cc:2454] T 9eb6f3c696db4acf8a048f8ddc9e1230 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:27.092535 17841 tablet_bootstrap.cc:492] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304: No bootstrap required, opened a new log
I20240502 01:55:27.092661 17841 ts_tablet_manager.cc:1367] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.009s
I20240502 01:55:27.092809 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb6f3c696db4acf8a048f8ddc9e1230" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:27.093027 16885 raft_consensus.cc:2454] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:27.093330 17837 tablet_bootstrap.cc:654] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:27.093307 17841 raft_consensus.cc:348] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.093640 17744 raft_consensus.cc:482] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:27.093662 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb6f3c696db4acf8a048f8ddc9e1230" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:27.093861 16225 raft_consensus.cc:2454] T 9eb6f3c696db4acf8a048f8ddc9e1230 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:27.093786 17744 raft_consensus.cc:504] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.094455 16689 leader_election.cc:302] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 6b9f9aaa1c86420190af75270914449a, a6575ebf978a4631a13ba5c243c3c4f9, cdd4d50250184223992325c86cb6c68e, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:27.094782 17827 raft_consensus.cc:2781] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:27.094893 17827 raft_consensus.cc:482] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:27.094967 17827 raft_consensus.cc:3037] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:27.095166 17744 leader_election.cc:288] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133)
I20240502 01:55:27.096792 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e676bcd8b04f18a6814d6ebbd1b488" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:27.096982 16225 raft_consensus.cc:2454] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:27.097147 17015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e676bcd8b04f18a6814d6ebbd1b488" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:27.097373 17015 raft_consensus.cc:2454] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:27.097365 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e676bcd8b04f18a6814d6ebbd1b488" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:27.097592 16754 raft_consensus.cc:2454] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:27.093565 17841 raft_consensus.cc:374] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:27.098174 17841 raft_consensus.cc:729] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Initialized, Role: FOLLOWER
I20240502 01:55:27.098188 16425 leader_election.cc:302] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, a6575ebf978a4631a13ba5c243c3c4f9, cdd4d50250184223992325c86cb6c68e, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:27.098402 17841 consensus_queue.cc:260] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.098667 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e676bcd8b04f18a6814d6ebbd1b488" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:27.098901 16885 raft_consensus.cc:2454] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:27.099635 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e676bcd8b04f18a6814d6ebbd1b488" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:27.099912 16619 raft_consensus.cc:2454] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:27.100598 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e676bcd8b04f18a6814d6ebbd1b488" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:27.100875 16359 raft_consensus.cc:2454] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:27.101827 17744 raft_consensus.cc:2781] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:27.101958 17744 raft_consensus.cc:482] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:27.102041 17744 raft_consensus.cc:3037] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:27.106746 17827 raft_consensus.cc:504] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:27.105049 17744 raft_consensus.cc:504] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.108691 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb6f3c696db4acf8a048f8ddc9e1230" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
I20240502 01:55:27.108965 16359 raft_consensus.cc:3037] T 9eb6f3c696db4acf8a048f8ddc9e1230 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:27.111642 17773 raft_consensus.cc:482] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:27.111850 17773 raft_consensus.cc:504] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.112398 16359 raft_consensus.cc:2454] T 9eb6f3c696db4acf8a048f8ddc9e1230 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:27.113162 17773 leader_election.cc:288] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:27.113373 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab1fc6db80c4e53a4f0cacccca74096" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:27.113639 16754 raft_consensus.cc:2454] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:27.113757 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab1fc6db80c4e53a4f0cacccca74096" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:27.113970 16490 raft_consensus.cc:2454] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:27.114584 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab1fc6db80c4e53a4f0cacccca74096" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:27.114853 16619 raft_consensus.cc:2454] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:27.114955 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab1fc6db80c4e53a4f0cacccca74096" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:27.115295 16952 leader_election.cc:302] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 703f6f043c6041a4b629933690580367, a6575ebf978a4631a13ba5c243c3c4f9, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
W20240502 01:55:27.115841 16950 leader_election.cc:341] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:27.115852 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab1fc6db80c4e53a4f0cacccca74096" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:27.116106 16225 raft_consensus.cc:2454] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:27.116818 17773 raft_consensus.cc:2781] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:27.116948 17773 raft_consensus.cc:482] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:27.117025 17773 raft_consensus.cc:3037] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:27.120128 17773 raft_consensus.cc:504] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.121589 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab1fc6db80c4e53a4f0cacccca74096" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367"
I20240502 01:55:27.121843 16619 raft_consensus.cc:3037] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:27.124914 16619 raft_consensus.cc:2454] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:27.114388 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab1fc6db80c4e53a4f0cacccca74096" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:27.125572 16359 raft_consensus.cc:2454] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:27.127553 16620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e676bcd8b04f18a6814d6ebbd1b488" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367"
I20240502 01:55:27.127775 16620 raft_consensus.cc:3037] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:27.128677 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab1fc6db80c4e53a4f0cacccca74096" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a"
W20240502 01:55:27.129273 16950 leader_election.cc:341] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:27.130640 16620 raft_consensus.cc:2454] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:27.131165 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab1fc6db80c4e53a4f0cacccca74096" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e"
I20240502 01:55:27.131419 16225 raft_consensus.cc:3037] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:27.131652 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb6f3c696db4acf8a048f8ddc9e1230" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304"
I20240502 01:55:27.131835 16490 raft_consensus.cc:3037] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:27.134171 16225 raft_consensus.cc:2454] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:27.135493 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab1fc6db80c4e53a4f0cacccca74096" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
I20240502 01:55:27.135881 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab1fc6db80c4e53a4f0cacccca74096" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad"
I20240502 01:55:27.136130 16754 raft_consensus.cc:3037] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:27.136328 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e676bcd8b04f18a6814d6ebbd1b488" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad"
I20240502 01:55:27.136557 16753 raft_consensus.cc:3037] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:27.138430 16490 raft_consensus.cc:2454] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:27.139655 16753 raft_consensus.cc:2454] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:27.139883 16754 raft_consensus.cc:2454] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:27.140264 16949 leader_election.cc:302] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 703f6f043c6041a4b629933690580367, a6575ebf978a4631a13ba5c243c3c4f9, cdd4d50250184223992325c86cb6c68e, ed2e7e4188d0400b8689e28bbf5096ad; no voters: 6b9f9aaa1c86420190af75270914449a
I20240502 01:55:27.140580 17714 raft_consensus.cc:2781] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:27.140728 17714 raft_consensus.cc:686] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Becoming Leader. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Running, Role: LEADER
I20240502 01:55:27.140720 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb6f3c696db4acf8a048f8ddc9e1230" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e"
I20240502 01:55:27.140933 16225 raft_consensus.cc:3037] T 9eb6f3c696db4acf8a048f8ddc9e1230 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:27.141125 17714 consensus_queue.cc:237] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.143625 16225 raft_consensus.cc:2454] T 9eb6f3c696db4acf8a048f8ddc9e1230 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:27.143620 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb6f3c696db4acf8a048f8ddc9e1230" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a"
I20240502 01:55:27.143890 16885 raft_consensus.cc:3037] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:27.144138 16689 leader_election.cc:302] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, cdd4d50250184223992325c86cb6c68e, dc9fcb8d12cb4ab28ec821c9473adeeb, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:27.144485 17698 raft_consensus.cc:2781] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:27.144618 17698 raft_consensus.cc:686] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Becoming Leader. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Running, Role: LEADER
I20240502 01:55:27.144949 17698 consensus_queue.cc:237] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:27.146909 16885 raft_consensus.cc:2454] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:27.148788 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e676bcd8b04f18a6814d6ebbd1b488" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a"
I20240502 01:55:27.149020 16885 raft_consensus.cc:3037] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:27.151679 16885 raft_consensus.cc:2454] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:27.152092 16424 leader_election.cc:302] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 6b9f9aaa1c86420190af75270914449a, 703f6f043c6041a4b629933690580367, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:27.158778 16359 raft_consensus.cc:3037] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:27.161708 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab1fc6db80c4e53a4f0cacccca74096" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304"
I20240502 01:55:27.161939 16490 raft_consensus.cc:3037] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:27.164819 16490 raft_consensus.cc:2454] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:27.166929 17827 leader_election.cc:288] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 election: Requested vote from peers dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:27.167052 16620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb6f3c696db4acf8a048f8ddc9e1230" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367"
I20240502 01:55:27.167121 17015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb6f3c696db4acf8a048f8ddc9e1230" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9"
I20240502 01:55:27.167281 16620 raft_consensus.cc:3037] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:27.167358 17015 raft_consensus.cc:3037] T 9eb6f3c696db4acf8a048f8ddc9e1230 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:27.167755 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e676bcd8b04f18a6814d6ebbd1b488" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9"
I20240502 01:55:27.167964 17016 raft_consensus.cc:3037] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:27.170099 17015 raft_consensus.cc:2454] T 9eb6f3c696db4acf8a048f8ddc9e1230 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:27.170228 16620 raft_consensus.cc:2454] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:27.170775 17016 raft_consensus.cc:2454] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:27.171908 17773 leader_election.cc:288] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [CANDIDATE]: Term 1 election: Requested vote from peers 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:27.173326 17744 leader_election.cc:288] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 election: Requested vote from peers 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133)
I20240502 01:55:27.174155 16081 catalog_manager.cc:5484] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: term changed from 0 to 1, leader changed from <none> to ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197). New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:27.175787 16358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e676bcd8b04f18a6814d6ebbd1b488" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
I20240502 01:55:27.176025 16358 raft_consensus.cc:3037] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:27.179059 16358 raft_consensus.cc:2454] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:27.180233 17744 raft_consensus.cc:2781] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:27.180420 17744 raft_consensus.cc:686] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Becoming Leader. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Running, Role: LEADER
I20240502 01:55:27.180809 17744 consensus_queue.cc:237] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.187690 16359 raft_consensus.cc:2454] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:27.196251 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18e676bcd8b04f18a6814d6ebbd1b488" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e"
I20240502 01:55:27.196538 16225 raft_consensus.cc:3037] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:27.196780 16081 catalog_manager.cc:5484] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199). New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:27.199733 16225 raft_consensus.cc:2454] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
W20240502 01:55:27.201562 16424 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.161.198:38121, user_credentials={real_user=slave}} blocked reactor thread for 49442.5us
I20240502 01:55:27.205302 17841 ts_tablet_manager.cc:1398] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304: Time spent starting tablet: real 0.113s user 0.024s sys 0.047s
I20240502 01:55:27.206233 16081 catalog_manager.cc:5484] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39059854b64045698ded1ba4e8d47304 (127.31.161.195). New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:27.211228 17837 tablet_bootstrap.cc:492] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a: No bootstrap required, opened a new log
I20240502 01:55:27.211401 17837 ts_tablet_manager.cc:1367] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a: Time spent bootstrapping tablet: real 0.120s user 0.035s sys 0.044s
I20240502 01:55:27.212181 17837 raft_consensus.cc:348] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.212466 17837 raft_consensus.cc:374] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:27.212564 17837 raft_consensus.cc:729] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Initialized, Role: FOLLOWER
I20240502 01:55:27.212836 17837 consensus_queue.cc:260] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.213487 17837 ts_tablet_manager.cc:1398] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 01:55:27.237828 17859 raft_consensus.cc:482] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:27.238013 17859 raft_consensus.cc:504] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:27.239327 17859 leader_election.cc:288] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:27.239373 16620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab1fc6db80c4e53a4f0cacccca74096" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:27.239671 16620 raft_consensus.cc:2379] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b9f9aaa1c86420190af75270914449a in current term 1: Already voted for candidate a6575ebf978a4631a13ba5c243c3c4f9 in this term.
I20240502 01:55:27.240016 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab1fc6db80c4e53a4f0cacccca74096" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:27.240056 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab1fc6db80c4e53a4f0cacccca74096" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:27.240314 16225 raft_consensus.cc:2379] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b9f9aaa1c86420190af75270914449a in current term 1: Already voted for candidate a6575ebf978a4631a13ba5c243c3c4f9 in this term.
I20240502 01:55:27.240495 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab1fc6db80c4e53a4f0cacccca74096" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:27.240741 16359 raft_consensus.cc:2379] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b9f9aaa1c86420190af75270914449a in current term 1: Already voted for candidate a6575ebf978a4631a13ba5c243c3c4f9 in this term.
I20240502 01:55:27.241627 16821 leader_election.cc:302] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 6b9f9aaa1c86420190af75270914449a; no voters: 703f6f043c6041a4b629933690580367, a6575ebf978a4631a13ba5c243c3c4f9, cdd4d50250184223992325c86cb6c68e, dc9fcb8d12cb4ab28ec821c9473adeeb
I20240502 01:55:27.241575 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab1fc6db80c4e53a4f0cacccca74096" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:27.241845 16490 raft_consensus.cc:2379] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b9f9aaa1c86420190af75270914449a in current term 1: Already voted for candidate a6575ebf978a4631a13ba5c243c3c4f9 in this term.
I20240502 01:55:27.241905 17859 raft_consensus.cc:3037] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:27.242214 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bab1fc6db80c4e53a4f0cacccca74096" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:27.242460 16754 raft_consensus.cc:2379] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6b9f9aaa1c86420190af75270914449a in current term 1: Already voted for candidate a6575ebf978a4631a13ba5c243c3c4f9 in this term.
I20240502 01:55:27.244246 17859 raft_consensus.cc:2726] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:55:27.532682 17714 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:27.545372 17714 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:27.564356 17870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:27.564731 17870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:55:27.572827 17698 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:27.576431 17870 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:55:27.586680 17714 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:27.592075 17714 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:27.592757 17773 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:27.597466 17864 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:27.602299 17698 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:27.619536 17773 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240502 01:55:27.675770 17864 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:27.678401 17698 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:27.686560 17698 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:27.712613 17864 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:27.750962 17864 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:27.766511 17864 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:27.769508 17864 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:27.841934 17921 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:27.859697 16687 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.195:35693, user_credentials={real_user=slave}} blocked reactor thread for 148377us
I20240502 01:55:27.870332 17915 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 615.001 ms
time per row: 0.307501 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 01:55:29.237339 16081 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58366:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 01:55:29.238065 16081 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_3 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240502 01:55:29.246891 16205 tablet_service.cc:1467] Processing CreateTablet for tablet 4da77826275d42b2b9e280c732575cfd (DEFAULT_TABLE table=rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.247179 16338 tablet_service.cc:1467] Processing CreateTablet for tablet 4da77826275d42b2b9e280c732575cfd (DEFAULT_TABLE table=rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.247411 16205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da77826275d42b2b9e280c732575cfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.247396 16471 tablet_service.cc:1467] Processing CreateTablet for tablet 4da77826275d42b2b9e280c732575cfd (DEFAULT_TABLE table=rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.247630 16338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da77826275d42b2b9e280c732575cfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.247804 16471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da77826275d42b2b9e280c732575cfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.250139 16733 tablet_service.cc:1467] Processing CreateTablet for tablet 4da77826275d42b2b9e280c732575cfd (DEFAULT_TABLE table=rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.250571 16733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da77826275d42b2b9e280c732575cfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.250662 16996 tablet_service.cc:1467] Processing CreateTablet for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c (DEFAULT_TABLE table=rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.251077 16996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.253237 16204 tablet_service.cc:1467] Processing CreateTablet for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c (DEFAULT_TABLE table=rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.254451 16995 tablet_service.cc:1467] Processing CreateTablet for tablet 4da77826275d42b2b9e280c732575cfd (DEFAULT_TABLE table=rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.254813 16995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da77826275d42b2b9e280c732575cfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.254992 16600 tablet_service.cc:1467] Processing CreateTablet for tablet 4da77826275d42b2b9e280c732575cfd (DEFAULT_TABLE table=rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.255290 16470 tablet_service.cc:1467] Processing CreateTablet for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c (DEFAULT_TABLE table=rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.255492 16600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da77826275d42b2b9e280c732575cfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.255671 16470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.254009 16732 tablet_service.cc:1467] Processing CreateTablet for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c (DEFAULT_TABLE table=rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.257479 16602 tablet_service.cc:1467] Processing CreateTablet for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c (DEFAULT_TABLE table=rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.257846 16732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.257887 16602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.261561 16865 tablet_service.cc:1467] Processing CreateTablet for tablet 4da77826275d42b2b9e280c732575cfd (DEFAULT_TABLE table=rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.262051 16865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da77826275d42b2b9e280c732575cfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.265064 16601 tablet_service.cc:1467] Processing CreateTablet for tablet cec84c633b0a4c4fb6e074c79432fe61 (DEFAULT_TABLE table=rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.265435 16601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec84c633b0a4c4fb6e074c79432fe61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.265908 16864 tablet_service.cc:1467] Processing CreateTablet for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c (DEFAULT_TABLE table=rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.266073 16339 tablet_service.cc:1467] Processing CreateTablet for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c (DEFAULT_TABLE table=rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.266268 16864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.266491 16339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.268787 16994 tablet_service.cc:1467] Processing CreateTablet for tablet cec84c633b0a4c4fb6e074c79432fe61 (DEFAULT_TABLE table=rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.269170 16994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec84c633b0a4c4fb6e074c79432fe61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.269419 16863 tablet_service.cc:1467] Processing CreateTablet for tablet cec84c633b0a4c4fb6e074c79432fe61 (DEFAULT_TABLE table=rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.270149 16204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.274961 16734 tablet_service.cc:1467] Processing CreateTablet for tablet cec84c633b0a4c4fb6e074c79432fe61 (DEFAULT_TABLE table=rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.275384 16734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec84c633b0a4c4fb6e074c79432fe61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.275744 16863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec84c633b0a4c4fb6e074c79432fe61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.277601 16470 tablet_service.cc:1467] Processing CreateTablet for tablet cec84c633b0a4c4fb6e074c79432fe61 (DEFAULT_TABLE table=rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.278017 16470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec84c633b0a4c4fb6e074c79432fe61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.281388 17981 tablet_bootstrap.cc:492] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:29.281512 16337 tablet_service.cc:1467] Processing CreateTablet for tablet cec84c633b0a4c4fb6e074c79432fe61 (DEFAULT_TABLE table=rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.281917 16337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec84c633b0a4c4fb6e074c79432fe61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.284148 17981 tablet_bootstrap.cc:654] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.285663 16203 tablet_service.cc:1467] Processing CreateTablet for tablet cec84c633b0a4c4fb6e074c79432fe61 (DEFAULT_TABLE table=rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:29.286108 16203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec84c633b0a4c4fb6e074c79432fe61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:29.290696 17980 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9: Bootstrap starting.
I20240502 01:55:29.292603 17981 tablet_bootstrap.cc:492] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367: No bootstrap required, opened a new log
I20240502 01:55:29.292774 17981 ts_tablet_manager.cc:1367] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.004s
I20240502 01:55:29.293303 17980 tablet_bootstrap.cc:654] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.296985 17981 raft_consensus.cc:348] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:29.297297 17981 raft_consensus.cc:374] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.297518 17981 raft_consensus.cc:729] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.301816 17981 consensus_queue.cc:260] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:29.303695 17981 ts_tablet_manager.cc:1398] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240502 01:55:29.309792 17981 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:29.315582 17981 tablet_bootstrap.cc:654] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.324451 17981 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367: No bootstrap required, opened a new log
I20240502 01:55:29.325992 17986 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:29.326238 17981 ts_tablet_manager.cc:1367] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240502 01:55:29.328573 17986 tablet_bootstrap.cc:654] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.327025 17981 raft_consensus.cc:348] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.330157 17981 raft_consensus.cc:374] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.330585 17981 raft_consensus.cc:729] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.331427 17981 consensus_queue.cc:260] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.333647 17981 ts_tablet_manager.cc:1398] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240502 01:55:29.334378 17981 tablet_bootstrap.cc:492] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:29.336409 17981 tablet_bootstrap.cc:654] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.336671 17986 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad: No bootstrap required, opened a new log
I20240502 01:55:29.336845 17986 ts_tablet_manager.cc:1367] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 01:55:29.337635 17986 raft_consensus.cc:348] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.337941 17986 raft_consensus.cc:374] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.338037 17986 raft_consensus.cc:729] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.338285 17986 consensus_queue.cc:260] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.342511 17986 ts_tablet_manager.cc:1398] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:55:29.343282 17986 tablet_bootstrap.cc:492] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:29.351225 17980 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9: No bootstrap required, opened a new log
I20240502 01:55:29.351438 17980 ts_tablet_manager.cc:1367] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9: Time spent bootstrapping tablet: real 0.061s user 0.018s sys 0.018s
I20240502 01:55:29.352260 17980 raft_consensus.cc:348] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.352563 17980 raft_consensus.cc:374] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.352653 17980 raft_consensus.cc:729] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.352982 17980 consensus_queue.cc:260] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.356993 17985 tablet_bootstrap.cc:492] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb: Bootstrap starting.
I20240502 01:55:29.357262 17980 ts_tablet_manager.cc:1398] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 01:55:29.358964 17986 tablet_bootstrap.cc:654] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.359084 17980 tablet_bootstrap.cc:492] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9: Bootstrap starting.
I20240502 01:55:29.359934 17985 tablet_bootstrap.cc:654] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.362051 17980 tablet_bootstrap.cc:654] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.362669 17987 tablet_bootstrap.cc:492] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e: Bootstrap starting.
I20240502 01:55:29.363008 17984 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a: Bootstrap starting.
I20240502 01:55:29.363710 17986 tablet_bootstrap.cc:492] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad: No bootstrap required, opened a new log
I20240502 01:55:29.363873 17986 ts_tablet_manager.cc:1367] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20240502 01:55:29.364513 17986 raft_consensus.cc:348] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:29.364686 17984 tablet_bootstrap.cc:654] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.364740 17986 raft_consensus.cc:374] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.364817 17986 raft_consensus.cc:729] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.365053 17986 consensus_queue.cc:260] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:29.365666 17986 ts_tablet_manager.cc:1398] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:29.366037 17986 tablet_bootstrap.cc:492] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:29.367849 17986 tablet_bootstrap.cc:654] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.368693 17980 tablet_bootstrap.cc:492] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9: No bootstrap required, opened a new log
I20240502 01:55:29.368862 17980 ts_tablet_manager.cc:1367] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 01:55:29.368999 17987 tablet_bootstrap.cc:654] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.369622 17980 raft_consensus.cc:348] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:29.370610 17980 raft_consensus.cc:374] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.370707 17980 raft_consensus.cc:729] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.370987 17980 consensus_queue.cc:260] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:29.372372 17980 ts_tablet_manager.cc:1398] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:55:29.372671 17980 tablet_bootstrap.cc:492] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9: Bootstrap starting.
I20240502 01:55:29.375802 17984 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a: No bootstrap required, opened a new log
I20240502 01:55:29.375949 17985 tablet_bootstrap.cc:492] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb: No bootstrap required, opened a new log
I20240502 01:55:29.376233 17985 ts_tablet_manager.cc:1367] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240502 01:55:29.377493 17984 ts_tablet_manager.cc:1367] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240502 01:55:29.378423 17980 tablet_bootstrap.cc:654] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.378536 17985 raft_consensus.cc:348] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:29.378895 17985 raft_consensus.cc:374] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.379021 17985 raft_consensus.cc:729] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fcb8d12cb4ab28ec821c9473adeeb, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.378345 17984 raft_consensus.cc:348] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.379521 17985 consensus_queue.cc:260] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:29.379854 17984 raft_consensus.cc:374] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.379992 17984 raft_consensus.cc:729] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.380309 17984 consensus_queue.cc:260] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.381582 17979 tablet_bootstrap.cc:492] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304: Bootstrap starting.
I20240502 01:55:29.382308 17987 tablet_bootstrap.cc:492] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e: No bootstrap required, opened a new log
I20240502 01:55:29.383268 17979 tablet_bootstrap.cc:654] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.383828 17987 ts_tablet_manager.cc:1367] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.004s
I20240502 01:55:29.386619 17987 raft_consensus.cc:348] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.387609 17981 tablet_bootstrap.cc:492] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367: No bootstrap required, opened a new log
I20240502 01:55:29.387782 17981 ts_tablet_manager.cc:1367] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.053s user 0.006s sys 0.006s
I20240502 01:55:29.388373 17987 raft_consensus.cc:374] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.388648 17981 raft_consensus.cc:348] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.389228 17981 raft_consensus.cc:374] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.388849 17987 raft_consensus.cc:729] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4d50250184223992325c86cb6c68e, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.389377 17981 raft_consensus.cc:729] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.389663 17981 consensus_queue.cc:260] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.390422 17981 ts_tablet_manager.cc:1398] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:55:29.391484 17985 ts_tablet_manager.cc:1398] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent starting tablet: real 0.015s user 0.002s sys 0.003s
I20240502 01:55:29.393569 17979 tablet_bootstrap.cc:492] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304: No bootstrap required, opened a new log
I20240502 01:55:29.393818 17979 ts_tablet_manager.cc:1367] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 01:55:29.389784 17987 consensus_queue.cc:260] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.394667 17979 raft_consensus.cc:348] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.395090 17979 raft_consensus.cc:374] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.395143 17985 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb: Bootstrap starting.
I20240502 01:55:29.395282 17979 raft_consensus.cc:729] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.395601 17979 consensus_queue.cc:260] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.395941 17984 ts_tablet_manager.cc:1398] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20240502 01:55:29.396246 17984 tablet_bootstrap.cc:492] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a: Bootstrap starting.
I20240502 01:55:29.396507 17979 ts_tablet_manager.cc:1398] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:55:29.396838 17979 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304: Bootstrap starting.
I20240502 01:55:29.398455 17979 tablet_bootstrap.cc:654] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.398509 17984 tablet_bootstrap.cc:654] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.403913 17979 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304: No bootstrap required, opened a new log
I20240502 01:55:29.404157 17985 tablet_bootstrap.cc:654] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.404153 17979 ts_tablet_manager.cc:1367] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 01:55:29.404882 17979 raft_consensus.cc:348] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.405241 17979 raft_consensus.cc:374] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.405332 17979 raft_consensus.cc:729] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.405413 17987 ts_tablet_manager.cc:1398] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240502 01:55:29.405736 17987 tablet_bootstrap.cc:492] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e: Bootstrap starting.
I20240502 01:55:29.405575 17979 consensus_queue.cc:260] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.406234 17979 ts_tablet_manager.cc:1398] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:55:29.407305 17987 tablet_bootstrap.cc:654] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.408056 17979 tablet_bootstrap.cc:492] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304: Bootstrap starting.
I20240502 01:55:29.409555 17979 tablet_bootstrap.cc:654] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.412088 17985 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb: No bootstrap required, opened a new log
I20240502 01:55:29.412277 17985 ts_tablet_manager.cc:1367] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.003s
I20240502 01:55:29.413224 17984 tablet_bootstrap.cc:492] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a: No bootstrap required, opened a new log
I20240502 01:55:29.413015 17985 raft_consensus.cc:348] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.413403 17985 raft_consensus.cc:374] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.413491 17985 raft_consensus.cc:729] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fcb8d12cb4ab28ec821c9473adeeb, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.413602 17980 tablet_bootstrap.cc:492] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9: No bootstrap required, opened a new log
I20240502 01:55:29.413540 17984 ts_tablet_manager.cc:1367] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240502 01:55:29.413763 17980 ts_tablet_manager.cc:1367] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.008s
I20240502 01:55:29.413786 17985 consensus_queue.cc:260] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.414505 17980 raft_consensus.cc:348] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.414780 17980 raft_consensus.cc:374] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.414880 17980 raft_consensus.cc:729] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.415148 17984 raft_consensus.cc:348] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.415129 17980 consensus_queue.cc:260] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.415396 17984 raft_consensus.cc:374] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.415462 17984 raft_consensus.cc:729] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.415731 17984 consensus_queue.cc:260] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.416604 17985 ts_tablet_manager.cc:1398] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 01:55:29.416639 17980 ts_tablet_manager.cc:1398] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:55:29.416924 17985 tablet_bootstrap.cc:492] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb: Bootstrap starting.
I20240502 01:55:29.418041 17984 ts_tablet_manager.cc:1398] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:55:29.418354 17984 tablet_bootstrap.cc:492] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a: Bootstrap starting.
I20240502 01:55:29.419925 17984 tablet_bootstrap.cc:654] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.421959 17985 tablet_bootstrap.cc:654] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.423599 17985 tablet_bootstrap.cc:492] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb: No bootstrap required, opened a new log
I20240502 01:55:29.423734 17985 ts_tablet_manager.cc:1367] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240502 01:55:29.424443 17985 raft_consensus.cc:348] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.424654 17985 raft_consensus.cc:374] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.424741 17985 raft_consensus.cc:729] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fcb8d12cb4ab28ec821c9473adeeb, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.424958 17985 consensus_queue.cc:260] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.425669 17985 ts_tablet_manager.cc:1398] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:29.427642 17986 tablet_bootstrap.cc:492] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad: No bootstrap required, opened a new log
I20240502 01:55:29.427834 17986 ts_tablet_manager.cc:1367] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.062s user 0.011s sys 0.010s
I20240502 01:55:29.428634 17986 raft_consensus.cc:348] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.428936 17986 raft_consensus.cc:374] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.429031 17986 raft_consensus.cc:729] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.429297 17986 consensus_queue.cc:260] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.429931 17986 ts_tablet_manager.cc:1398] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:55:29.430567 17979 tablet_bootstrap.cc:492] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304: No bootstrap required, opened a new log
I20240502 01:55:29.430763 17979 ts_tablet_manager.cc:1367] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20240502 01:55:29.431537 17979 raft_consensus.cc:348] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:29.431798 17979 raft_consensus.cc:374] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.431874 17979 raft_consensus.cc:729] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.432101 17979 consensus_queue.cc:260] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:29.433017 17979 ts_tablet_manager.cc:1398] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:29.442225 17987 tablet_bootstrap.cc:492] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e: No bootstrap required, opened a new log
I20240502 01:55:29.442415 17987 ts_tablet_manager.cc:1367] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e: Time spent bootstrapping tablet: real 0.037s user 0.015s sys 0.007s
I20240502 01:55:29.443141 17987 raft_consensus.cc:348] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:29.443410 17987 raft_consensus.cc:374] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.443485 17987 raft_consensus.cc:729] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4d50250184223992325c86cb6c68e, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.443758 17987 consensus_queue.cc:260] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:29.444461 17987 ts_tablet_manager.cc:1398] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 01:55:29.444753 17987 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e: Bootstrap starting.
I20240502 01:55:29.446288 17987 tablet_bootstrap.cc:654] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:29.447866 17973 raft_consensus.cc:482] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:29.448062 17973 raft_consensus.cc:504] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.448226 17813 raft_consensus.cc:482] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:29.448410 17813 raft_consensus.cc:504] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:29.449342 17973 leader_election.cc:288] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:29.449406 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6150a46aa8ca4d9bbd16c141ad2f1a3c" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:29.450251 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6150a46aa8ca4d9bbd16c141ad2f1a3c" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:29.450516 16359 raft_consensus.cc:2454] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:29.450632 16620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6150a46aa8ca4d9bbd16c141ad2f1a3c" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:29.450868 16620 raft_consensus.cc:2454] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:29.451022 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6150a46aa8ca4d9bbd16c141ad2f1a3c" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:29.451256 16753 raft_consensus.cc:2454] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:29.450986 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec84c633b0a4c4fb6e074c79432fe61" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:29.451571 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6150a46aa8ca4d9bbd16c141ad2f1a3c" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:29.451645 16754 raft_consensus.cc:2454] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:29.451807 16885 raft_consensus.cc:2454] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:29.452340 16426 leader_election.cc:302] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 6b9f9aaa1c86420190af75270914449a, dc9fcb8d12cb4ab28ec821c9473adeeb, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:29.452885 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6150a46aa8ca4d9bbd16c141ad2f1a3c" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:29.453157 17016 raft_consensus.cc:2454] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:29.453310 17813 leader_election.cc:288] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:29.453701 17984 tablet_bootstrap.cc:492] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a: No bootstrap required, opened a new log
I20240502 01:55:29.453856 17984 ts_tablet_manager.cc:1367] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20240502 01:55:29.454026 16224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec84c633b0a4c4fb6e074c79432fe61" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:29.454239 16224 raft_consensus.cc:2454] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:29.454272 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec84c633b0a4c4fb6e074c79432fe61" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:29.454470 16359 raft_consensus.cc:2454] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:29.454664 17984 raft_consensus.cc:348] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:29.454855 16558 leader_election.cc:302] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 703f6f043c6041a4b629933690580367, cdd4d50250184223992325c86cb6c68e, dc9fcb8d12cb4ab28ec821c9473adeeb, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:29.454932 17984 raft_consensus.cc:374] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.455027 17984 raft_consensus.cc:729] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.455139 17813 raft_consensus.cc:2781] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:29.455247 17813 raft_consensus.cc:482] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:29.455329 17813 raft_consensus.cc:3037] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.455263 17984 consensus_queue.cc:260] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:29.455410 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec84c633b0a4c4fb6e074c79432fe61" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:29.455457 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec84c633b0a4c4fb6e074c79432fe61" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:29.455595 16885 raft_consensus.cc:2454] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:29.455658 16490 raft_consensus.cc:2454] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:29.455952 17984 ts_tablet_manager.cc:1398] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:55:29.457036 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec84c633b0a4c4fb6e074c79432fe61" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:29.457334 17016 raft_consensus.cc:2454] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:29.457648 17973 raft_consensus.cc:2781] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:29.457759 17973 raft_consensus.cc:482] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:29.457836 17973 raft_consensus.cc:3037] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.458527 17813 raft_consensus.cc:504] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:29.459676 17813 leader_election.cc:288] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 election: Requested vote from peers ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:29.460637 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec84c633b0a4c4fb6e074c79432fe61" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad"
I20240502 01:55:29.460860 16754 raft_consensus.cc:3037] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.460850 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec84c633b0a4c4fb6e074c79432fe61" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304"
I20240502 01:55:29.461038 16490 raft_consensus.cc:3037] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.463032 16490 raft_consensus.cc:2454] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
I20240502 01:55:29.463645 16754 raft_consensus.cc:2454] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
I20240502 01:55:29.463704 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec84c633b0a4c4fb6e074c79432fe61" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
I20240502 01:55:29.463900 16359 raft_consensus.cc:3037] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.464195 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec84c633b0a4c4fb6e074c79432fe61" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a"
I20240502 01:55:29.464417 16885 raft_consensus.cc:3037] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.466055 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec84c633b0a4c4fb6e074c79432fe61" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9"
I20240502 01:55:29.466307 17016 raft_consensus.cc:3037] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.466354 17973 raft_consensus.cc:504] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.466553 16359 raft_consensus.cc:2454] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
I20240502 01:55:29.467001 16558 leader_election.cc:302] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 703f6f043c6041a4b629933690580367, dc9fcb8d12cb4ab28ec821c9473adeeb, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:29.467545 17973 leader_election.cc:288] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 election: Requested vote from peers cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:29.467705 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6150a46aa8ca4d9bbd16c141ad2f1a3c" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad"
I20240502 01:55:29.467932 16753 raft_consensus.cc:3037] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.467880 16224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6150a46aa8ca4d9bbd16c141ad2f1a3c" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e"
I20240502 01:55:29.468528 17987 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e: No bootstrap required, opened a new log
I20240502 01:55:29.468689 17987 ts_tablet_manager.cc:1367] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240502 01:55:29.468816 17016 raft_consensus.cc:2454] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
W20240502 01:55:29.468827 16425 leader_election.cc:341] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:55:29.469134 16425 leader_election.cc:341] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:29.469486 17987 raft_consensus.cc:348] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.469760 17987 raft_consensus.cc:374] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:29.469843 17987 raft_consensus.cc:729] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4d50250184223992325c86cb6c68e, State: Initialized, Role: FOLLOWER
I20240502 01:55:29.470075 17987 consensus_queue.cc:260] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.470499 16753 raft_consensus.cc:2454] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:29.470885 16884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6150a46aa8ca4d9bbd16c141ad2f1a3c" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a"
I20240502 01:55:29.471096 16884 raft_consensus.cc:3037] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.471705 16885 raft_consensus.cc:2454] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
I20240502 01:55:29.472148 17987 ts_tablet_manager.cc:1398] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:55:29.472908 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec84c633b0a4c4fb6e074c79432fe61" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e"
I20240502 01:55:29.473117 16225 raft_consensus.cc:3037] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.473690 16884 raft_consensus.cc:2454] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:29.474563 17015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6150a46aa8ca4d9bbd16c141ad2f1a3c" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9"
I20240502 01:55:29.474781 17015 raft_consensus.cc:3037] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.475736 16225 raft_consensus.cc:2454] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
I20240502 01:55:29.478302 16358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6150a46aa8ca4d9bbd16c141ad2f1a3c" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
I20240502 01:55:29.478578 16358 raft_consensus.cc:3037] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.478595 16620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6150a46aa8ca4d9bbd16c141ad2f1a3c" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367"
I20240502 01:55:29.478806 16620 raft_consensus.cc:3037] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.481379 16358 raft_consensus.cc:2454] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:29.481959 18011 raft_consensus.cc:2781] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:29.482511 16426 leader_election.cc:302] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 6b9f9aaa1c86420190af75270914449a, dc9fcb8d12cb4ab28ec821c9473adeeb, ed2e7e4188d0400b8689e28bbf5096ad; no voters: cdd4d50250184223992325c86cb6c68e
I20240502 01:55:29.482903 17973 raft_consensus.cc:2781] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:29.483183 17973 raft_consensus.cc:686] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Becoming Leader. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Running, Role: LEADER
I20240502 01:55:29.483541 17973 consensus_queue.cc:237] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.484604 16620 raft_consensus.cc:2454] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:29.489226 17015 raft_consensus.cc:2454] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:29.488467 16077 catalog_manager.cc:5484] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39059854b64045698ded1ba4e8d47304 (127.31.161.195). New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:29.489995 18011 raft_consensus.cc:686] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Becoming Leader. State: Replica: 703f6f043c6041a4b629933690580367, State: Running, Role: LEADER
I20240502 01:55:29.490327 18011 consensus_queue.cc:237] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:29.494498 16075 catalog_manager.cc:5484] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 reported cstate change: term changed from 0 to 1, leader changed from <none> to 703f6f043c6041a4b629933690580367 (127.31.161.196). New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:29.573933 18000 raft_consensus.cc:482] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:29.574146 18000 raft_consensus.cc:504] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.575452 18000 leader_election.cc:288] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:29.576470 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da77826275d42b2b9e280c732575cfd" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:29.576766 16490 raft_consensus.cc:2454] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 0.
I20240502 01:55:29.576930 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da77826275d42b2b9e280c732575cfd" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:29.577219 16225 raft_consensus.cc:2454] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 0.
I20240502 01:55:29.578433 17015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da77826275d42b2b9e280c732575cfd" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:29.578475 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da77826275d42b2b9e280c732575cfd" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:29.578677 17015 raft_consensus.cc:2454] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 0.
I20240502 01:55:29.578754 16359 raft_consensus.cc:2454] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 0.
I20240502 01:55:29.579119 16818 leader_election.cc:302] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 6b9f9aaa1c86420190af75270914449a, a6575ebf978a4631a13ba5c243c3c4f9, cdd4d50250184223992325c86cb6c68e; no voters:
I20240502 01:55:29.579454 18000 raft_consensus.cc:2781] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:29.579591 18000 raft_consensus.cc:482] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:29.579732 18000 raft_consensus.cc:3037] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.581672 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da77826275d42b2b9e280c732575cfd" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:29.581923 16754 raft_consensus.cc:2454] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 0.
I20240502 01:55:29.582657 16620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da77826275d42b2b9e280c732575cfd" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:29.582652 18000 raft_consensus.cc:504] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.582927 16620 raft_consensus.cc:2454] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 0.
I20240502 01:55:29.584509 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da77826275d42b2b9e280c732575cfd" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e"
I20240502 01:55:29.584607 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da77826275d42b2b9e280c732575cfd" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
I20240502 01:55:29.584772 16359 raft_consensus.cc:3037] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.584746 16225 raft_consensus.cc:3037] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.586637 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da77826275d42b2b9e280c732575cfd" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304"
I20240502 01:55:29.586881 16490 raft_consensus.cc:3037] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.587484 16225 raft_consensus.cc:2454] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 1.
I20240502 01:55:29.587498 16359 raft_consensus.cc:2454] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 1.
I20240502 01:55:29.588596 18000 leader_election.cc:288] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [CANDIDATE]: Term 1 election: Requested vote from peers cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:29.589221 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da77826275d42b2b9e280c732575cfd" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad"
I20240502 01:55:29.589187 17015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da77826275d42b2b9e280c732575cfd" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9"
I20240502 01:55:29.589435 17015 raft_consensus.cc:3037] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.589469 16754 raft_consensus.cc:3037] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.590070 16490 raft_consensus.cc:2454] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 1.
I20240502 01:55:29.591998 17015 raft_consensus.cc:2454] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 1.
I20240502 01:55:29.592103 16754 raft_consensus.cc:2454] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 1.
I20240502 01:55:29.588945 16620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da77826275d42b2b9e280c732575cfd" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367"
I20240502 01:55:29.592418 16818 leader_election.cc:302] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 6b9f9aaa1c86420190af75270914449a, a6575ebf978a4631a13ba5c243c3c4f9, cdd4d50250184223992325c86cb6c68e, dc9fcb8d12cb4ab28ec821c9473adeeb; no voters:
I20240502 01:55:29.592533 16620 raft_consensus.cc:3037] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.592999 18000 raft_consensus.cc:2781] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:29.594693 18000 raft_consensus.cc:686] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [term 1 LEADER]: Becoming Leader. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Running, Role: LEADER
I20240502 01:55:29.595189 18000 consensus_queue.cc:237] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.595381 16620 raft_consensus.cc:2454] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 1.
I20240502 01:55:29.599409 16075 catalog_manager.cc:5484] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b9f9aaa1c86420190af75270914449a (127.31.161.198). New cstate: current_term: 1 leader_uuid: "6b9f9aaa1c86420190af75270914449a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:29.882634 17973 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:29.928475 18013 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:29.928841 18011 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:29.947309 17973 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:29.958175 17973 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:29.963917 17973 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:29.965993 16225 raft_consensus.cc:3037] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:29.976465 18002 raft_consensus.cc:482] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:29.976660 18002 raft_consensus.cc:504] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:29.978282 18002 leader_election.cc:288] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:29.980346 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6150a46aa8ca4d9bbd16c141ad2f1a3c" candidate_uuid: "cdd4d50250184223992325c86cb6c68e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:29.980917 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6150a46aa8ca4d9bbd16c141ad2f1a3c" candidate_uuid: "cdd4d50250184223992325c86cb6c68e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:29.983882 16358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6150a46aa8ca4d9bbd16c141ad2f1a3c" candidate_uuid: "cdd4d50250184223992325c86cb6c68e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:29.985164 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6150a46aa8ca4d9bbd16c141ad2f1a3c" candidate_uuid: "cdd4d50250184223992325c86cb6c68e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:29.986027 18016 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:29.988214 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6150a46aa8ca4d9bbd16c141ad2f1a3c" candidate_uuid: "cdd4d50250184223992325c86cb6c68e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:29.989037 18000 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:29.990506 16491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6150a46aa8ca4d9bbd16c141ad2f1a3c" candidate_uuid: "cdd4d50250184223992325c86cb6c68e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:29.990612 16161 leader_election.cc:302] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: cdd4d50250184223992325c86cb6c68e; no voters: 6b9f9aaa1c86420190af75270914449a, 703f6f043c6041a4b629933690580367, a6575ebf978a4631a13ba5c243c3c4f9, dc9fcb8d12cb4ab28ec821c9473adeeb
I20240502 01:55:29.992334 18029 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:29.995352 18002 raft_consensus.cc:2726] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 01:55:29.998353 17990 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:30.027417 18011 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:30.028772 18018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:30.029191 18018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:55:30.033401 18013 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:30.040946 18018 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:55:30.050994 18011 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:30.051647 18011 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:30.058707 18000 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:30.072000 18016 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:30.087181 18016 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:30.087666 18016 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 470.383 ms
time per row: 0.235191 ms
I20240502 01:55:30.704517 16075 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58366:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 01:55:30.705369 16075 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_4 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240502 01:55:30.715272 16732 tablet_service.cc:1467] Processing CreateTablet for tablet 6cde24f949334006aa9e241e844e81eb (DEFAULT_TABLE table=rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.715392 16864 tablet_service.cc:1467] Processing CreateTablet for tablet 6cde24f949334006aa9e241e844e81eb (DEFAULT_TABLE table=rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.715759 16732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cde24f949334006aa9e241e844e81eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.715837 16864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cde24f949334006aa9e241e844e81eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.718364 16470 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa6224f34ef4bc58797f6cf801ff6da (DEFAULT_TABLE table=rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.718336 16995 tablet_service.cc:1467] Processing CreateTablet for tablet 6cde24f949334006aa9e241e844e81eb (DEFAULT_TABLE table=rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.718771 16995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cde24f949334006aa9e241e844e81eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.718801 16470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa6224f34ef4bc58797f6cf801ff6da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.718842 16205 tablet_service.cc:1467] Processing CreateTablet for tablet 6cde24f949334006aa9e241e844e81eb (DEFAULT_TABLE table=rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.719332 16205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cde24f949334006aa9e241e844e81eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.719427 16471 tablet_service.cc:1467] Processing CreateTablet for tablet 6cde24f949334006aa9e241e844e81eb (DEFAULT_TABLE table=rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.720153 16600 tablet_service.cc:1467] Processing CreateTablet for tablet 6cde24f949334006aa9e241e844e81eb (DEFAULT_TABLE table=rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.720614 16600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cde24f949334006aa9e241e844e81eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.721510 16204 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa6224f34ef4bc58797f6cf801ff6da (DEFAULT_TABLE table=rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.721895 16204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa6224f34ef4bc58797f6cf801ff6da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.725409 16338 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa6224f34ef4bc58797f6cf801ff6da (DEFAULT_TABLE table=rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.725826 16338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa6224f34ef4bc58797f6cf801ff6da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.729220 16733 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa6224f34ef4bc58797f6cf801ff6da (DEFAULT_TABLE table=rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.729609 16733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa6224f34ef4bc58797f6cf801ff6da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.731225 16602 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa6224f34ef4bc58797f6cf801ff6da (DEFAULT_TABLE table=rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.731714 16602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa6224f34ef4bc58797f6cf801ff6da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.733407 16337 tablet_service.cc:1467] Processing CreateTablet for tablet 6cde24f949334006aa9e241e844e81eb (DEFAULT_TABLE table=rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.733840 16337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cde24f949334006aa9e241e844e81eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.739013 16863 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa6224f34ef4bc58797f6cf801ff6da (DEFAULT_TABLE table=rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.740496 16863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa6224f34ef4bc58797f6cf801ff6da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.740869 16996 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa6224f34ef4bc58797f6cf801ff6da (DEFAULT_TABLE table=rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.741324 16996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa6224f34ef4bc58797f6cf801ff6da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.741458 16471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cde24f949334006aa9e241e844e81eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.749392 16601 tablet_service.cc:1467] Processing CreateTablet for tablet 53e266d2e3b446c3a50ed3b93a6a5dde (DEFAULT_TABLE table=rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.749868 16601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53e266d2e3b446c3a50ed3b93a6a5dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.761211 18132 tablet_bootstrap.cc:492] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:30.764103 18132 tablet_bootstrap.cc:654] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.764227 16865 tablet_service.cc:1467] Processing CreateTablet for tablet 53e266d2e3b446c3a50ed3b93a6a5dde (DEFAULT_TABLE table=rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.764222 16733 tablet_service.cc:1467] Processing CreateTablet for tablet 53e266d2e3b446c3a50ed3b93a6a5dde (DEFAULT_TABLE table=rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.764631 16865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53e266d2e3b446c3a50ed3b93a6a5dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.764652 16733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53e266d2e3b446c3a50ed3b93a6a5dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.767320 18129 tablet_bootstrap.cc:492] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a: Bootstrap starting.
I20240502 01:55:30.772028 16339 tablet_service.cc:1467] Processing CreateTablet for tablet 53e266d2e3b446c3a50ed3b93a6a5dde (DEFAULT_TABLE table=rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.772500 16339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53e266d2e3b446c3a50ed3b93a6a5dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.772431 16469 tablet_service.cc:1467] Processing CreateTablet for tablet 53e266d2e3b446c3a50ed3b93a6a5dde (DEFAULT_TABLE table=rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.772858 16469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53e266d2e3b446c3a50ed3b93a6a5dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.777499 16204 tablet_service.cc:1467] Processing CreateTablet for tablet 53e266d2e3b446c3a50ed3b93a6a5dde (DEFAULT_TABLE table=rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.778508 16204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53e266d2e3b446c3a50ed3b93a6a5dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.778954 18129 tablet_bootstrap.cc:654] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.781350 16996 tablet_service.cc:1467] Processing CreateTablet for tablet 53e266d2e3b446c3a50ed3b93a6a5dde (DEFAULT_TABLE table=rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:30.781819 16996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53e266d2e3b446c3a50ed3b93a6a5dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:30.786044 18133 tablet_bootstrap.cc:492] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e: Bootstrap starting.
I20240502 01:55:30.787719 18133 tablet_bootstrap.cc:654] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.788033 18132 tablet_bootstrap.cc:492] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367: No bootstrap required, opened a new log
I20240502 01:55:30.788308 18132 ts_tablet_manager.cc:1367] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.003s
I20240502 01:55:30.789216 18132 raft_consensus.cc:348] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.789697 18132 raft_consensus.cc:374] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.789777 18130 tablet_bootstrap.cc:492] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:30.789887 18132 raft_consensus.cc:729] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.790230 18132 consensus_queue.cc:260] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.791241 18131 tablet_bootstrap.cc:492] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9: Bootstrap starting.
I20240502 01:55:30.791549 18130 tablet_bootstrap.cc:654] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.791842 18132 ts_tablet_manager.cc:1398] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:55:30.792172 18132 tablet_bootstrap.cc:492] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:30.793534 18135 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb: Bootstrap starting.
I20240502 01:55:30.794502 18131 tablet_bootstrap.cc:654] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.798218 18138 tablet_bootstrap.cc:492] T 53e266d2e3b446c3a50ed3b93a6a5dde P 39059854b64045698ded1ba4e8d47304: Bootstrap starting.
I20240502 01:55:30.795331 18135 tablet_bootstrap.cc:654] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.800143 18129 tablet_bootstrap.cc:492] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a: No bootstrap required, opened a new log
I20240502 01:55:30.800395 18129 ts_tablet_manager.cc:1367] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20240502 01:55:30.801584 18129 raft_consensus.cc:348] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.801980 18129 raft_consensus.cc:374] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.802119 18129 raft_consensus.cc:729] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.802431 18129 consensus_queue.cc:260] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.803122 18133 tablet_bootstrap.cc:492] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e: No bootstrap required, opened a new log
I20240502 01:55:30.803288 18133 ts_tablet_manager.cc:1367] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.007s
I20240502 01:55:30.803550 18129 ts_tablet_manager.cc:1398] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:55:30.804113 18133 raft_consensus.cc:348] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.804451 18133 raft_consensus.cc:374] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.804538 18133 raft_consensus.cc:729] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4d50250184223992325c86cb6c68e, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.805300 18133 consensus_queue.cc:260] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.806489 18129 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a: Bootstrap starting.
I20240502 01:55:30.808223 18129 tablet_bootstrap.cc:654] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.810197 18132 tablet_bootstrap.cc:654] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.810492 18130 tablet_bootstrap.cc:492] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad: No bootstrap required, opened a new log
I20240502 01:55:30.811519 18130 ts_tablet_manager.cc:1367] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.009s
I20240502 01:55:30.811609 18138 tablet_bootstrap.cc:654] T 53e266d2e3b446c3a50ed3b93a6a5dde P 39059854b64045698ded1ba4e8d47304: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.812086 18131 tablet_bootstrap.cc:492] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9: No bootstrap required, opened a new log
I20240502 01:55:30.812255 18131 ts_tablet_manager.cc:1367] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.000s
I20240502 01:55:30.812410 18130 raft_consensus.cc:348] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.812847 18130 raft_consensus.cc:374] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.812980 18130 raft_consensus.cc:729] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.813292 18130 consensus_queue.cc:260] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.814110 18130 ts_tablet_manager.cc:1398] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:30.814955 18130 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:30.816605 18131 raft_consensus.cc:348] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.816920 18131 raft_consensus.cc:374] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.817088 18130 tablet_bootstrap.cc:654] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.817588 18135 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb: No bootstrap required, opened a new log
I20240502 01:55:30.817809 18135 ts_tablet_manager.cc:1367] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240502 01:55:30.819522 18135 raft_consensus.cc:348] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:30.819945 18135 raft_consensus.cc:374] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.820091 18135 raft_consensus.cc:729] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fcb8d12cb4ab28ec821c9473adeeb, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.820408 18135 consensus_queue.cc:260] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:30.817025 18131 raft_consensus.cc:729] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.821062 18131 consensus_queue.cc:260] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.822273 18132 tablet_bootstrap.cc:492] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367: No bootstrap required, opened a new log
I20240502 01:55:30.823215 18129 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a: No bootstrap required, opened a new log
I20240502 01:55:30.823453 18129 ts_tablet_manager.cc:1367] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240502 01:55:30.823812 18131 ts_tablet_manager.cc:1398] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:55:30.824146 18131 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9: Bootstrap starting.
I20240502 01:55:30.824469 18132 ts_tablet_manager.cc:1367] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.001s
I20240502 01:55:30.824455 18129 raft_consensus.cc:348] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:30.825300 18132 raft_consensus.cc:348] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.826105 18132 raft_consensus.cc:374] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.826051 18129 raft_consensus.cc:374] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.826191 18132 raft_consensus.cc:729] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.826296 18129 raft_consensus.cc:729] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.827373 18138 tablet_bootstrap.cc:492] T 53e266d2e3b446c3a50ed3b93a6a5dde P 39059854b64045698ded1ba4e8d47304: No bootstrap required, opened a new log
I20240502 01:55:30.827584 18138 ts_tablet_manager.cc:1367] T 53e266d2e3b446c3a50ed3b93a6a5dde P 39059854b64045698ded1ba4e8d47304: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.000s
I20240502 01:55:30.828125 18131 tablet_bootstrap.cc:654] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.828405 18129 consensus_queue.cc:260] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:30.828553 18132 consensus_queue.cc:260] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.828963 18138 raft_consensus.cc:348] T 53e266d2e3b446c3a50ed3b93a6a5dde P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.829883 18138 raft_consensus.cc:374] T 53e266d2e3b446c3a50ed3b93a6a5dde P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.830036 18138 raft_consensus.cc:729] T 53e266d2e3b446c3a50ed3b93a6a5dde P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.830152 18129 ts_tablet_manager.cc:1398] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:55:30.830610 18129 tablet_bootstrap.cc:492] T 53e266d2e3b446c3a50ed3b93a6a5dde P 6b9f9aaa1c86420190af75270914449a: Bootstrap starting.
I20240502 01:55:30.830833 18138 consensus_queue.cc:260] T 53e266d2e3b446c3a50ed3b93a6a5dde P 39059854b64045698ded1ba4e8d47304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.831151 18044 raft_consensus.cc:482] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:30.831302 18044 raft_consensus.cc:504] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.832036 18138 ts_tablet_manager.cc:1398] T 53e266d2e3b446c3a50ed3b93a6a5dde P 39059854b64045698ded1ba4e8d47304: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:55:30.832556 18044 leader_election.cc:288] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:30.833652 18138 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304: Bootstrap starting.
I20240502 01:55:30.833935 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cde24f949334006aa9e241e844e81eb" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:30.834172 16753 raft_consensus.cc:2454] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:30.834514 18129 tablet_bootstrap.cc:654] T 53e266d2e3b446c3a50ed3b93a6a5dde P 6b9f9aaa1c86420190af75270914449a: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.834720 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cde24f949334006aa9e241e844e81eb" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:30.834933 17016 raft_consensus.cc:2454] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:30.835317 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cde24f949334006aa9e241e844e81eb" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:30.836503 16224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cde24f949334006aa9e241e844e81eb" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:30.836777 16224 raft_consensus.cc:2454] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:30.837615 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cde24f949334006aa9e241e844e81eb" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:30.838351 18130 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad: No bootstrap required, opened a new log
I20240502 01:55:30.838420 18138 tablet_bootstrap.cc:654] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.838529 18130 ts_tablet_manager.cc:1367] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.010s
I20240502 01:55:30.839354 18130 raft_consensus.cc:348] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:30.839685 18130 raft_consensus.cc:374] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.839653 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cde24f949334006aa9e241e844e81eb" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:30.839751 18130 raft_consensus.cc:729] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.839879 16885 raft_consensus.cc:2454] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:30.840015 18130 consensus_queue.cc:260] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:30.840728 18130 ts_tablet_manager.cc:1398] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 01:55:30.841050 18130 tablet_bootstrap.cc:492] T 53e266d2e3b446c3a50ed3b93a6a5dde P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:30.841399 18133 ts_tablet_manager.cc:1398] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e: Time spent starting tablet: real 0.038s user 0.005s sys 0.022s
I20240502 01:55:30.841709 18133 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e: Bootstrap starting.
W20240502 01:55:30.841965 16555 leader_election.cc:341] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:55:30.842901 16558 leader_election.cc:341] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:30.843560 18133 tablet_bootstrap.cc:654] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.844274 18130 tablet_bootstrap.cc:654] T 53e266d2e3b446c3a50ed3b93a6a5dde P ed2e7e4188d0400b8689e28bbf5096ad: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.845997 16556 leader_election.cc:302] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 6b9f9aaa1c86420190af75270914449a, 703f6f043c6041a4b629933690580367, a6575ebf978a4631a13ba5c243c3c4f9, cdd4d50250184223992325c86cb6c68e; no voters: 39059854b64045698ded1ba4e8d47304, dc9fcb8d12cb4ab28ec821c9473adeeb
I20240502 01:55:30.848575 18138 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304: No bootstrap required, opened a new log
I20240502 01:55:30.848762 18138 ts_tablet_manager.cc:1367] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240502 01:55:30.852537 18044 raft_consensus.cc:2781] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:30.852665 18044 raft_consensus.cc:482] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:30.852767 18044 raft_consensus.cc:3037] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.852641 18138 raft_consensus.cc:348] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:30.852923 18138 raft_consensus.cc:374] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.852999 18138 raft_consensus.cc:729] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.854481 18138 consensus_queue.cc:260] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:30.855549 18044 raft_consensus.cc:504] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.856808 18138 ts_tablet_manager.cc:1398] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20240502 01:55:30.857141 18138 tablet_bootstrap.cc:492] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304: Bootstrap starting.
I20240502 01:55:30.858691 18138 tablet_bootstrap.cc:654] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.858923 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cde24f949334006aa9e241e844e81eb" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad"
I20240502 01:55:30.859172 16753 raft_consensus.cc:3037] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.860491 18117 raft_consensus.cc:482] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:30.860647 18117 raft_consensus.cc:504] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:30.861994 18117 leader_election.cc:288] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:30.863085 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa6224f34ef4bc58797f6cf801ff6da" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
W20240502 01:55:30.863858 16423 leader_election.cc:341] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:30.864116 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa6224f34ef4bc58797f6cf801ff6da" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:30.864357 16359 raft_consensus.cc:2454] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:30.864461 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa6224f34ef4bc58797f6cf801ff6da" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:30.864681 16754 raft_consensus.cc:2454] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:30.864917 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa6224f34ef4bc58797f6cf801ff6da" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:30.865026 16224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa6224f34ef4bc58797f6cf801ff6da" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
W20240502 01:55:30.865620 16425 leader_election.cc:341] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:55:30.865897 16426 leader_election.cc:341] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 703f6f043c6041a4b629933690580367 (127.31.161.196:40425): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:30.866431 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa6224f34ef4bc58797f6cf801ff6da" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:30.866662 16885 raft_consensus.cc:2454] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:30.867121 16424 leader_election.cc:302] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 7 responses out of 7 voters: 4 yes votes; 3 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 6b9f9aaa1c86420190af75270914449a, dc9fcb8d12cb4ab28ec821c9473adeeb, ed2e7e4188d0400b8689e28bbf5096ad; no voters: 703f6f043c6041a4b629933690580367, a6575ebf978a4631a13ba5c243c3c4f9, cdd4d50250184223992325c86cb6c68e
I20240502 01:55:30.867691 18117 raft_consensus.cc:2781] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:30.867799 18117 raft_consensus.cc:482] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:30.867868 18117 raft_consensus.cc:3037] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.870669 18117 raft_consensus.cc:504] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:30.872112 16224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa6224f34ef4bc58797f6cf801ff6da" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e"
I20240502 01:55:30.872560 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa6224f34ef4bc58797f6cf801ff6da" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367"
I20240502 01:55:30.872709 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa6224f34ef4bc58797f6cf801ff6da" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad"
I20240502 01:55:30.872910 16754 raft_consensus.cc:3037] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.873584 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa6224f34ef4bc58797f6cf801ff6da" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
I20240502 01:55:30.873782 16359 raft_consensus.cc:3037] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.875692 16754 raft_consensus.cc:2454] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:30.876497 16359 raft_consensus.cc:2454] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
W20240502 01:55:30.877022 16425 leader_election.cc:341] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953): Illegal state: must be running to vote when last-logged opid is not known
W20240502 01:55:30.877468 16426 leader_election.cc:341] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 703f6f043c6041a4b629933690580367 (127.31.161.196:40425): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:30.877748 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa6224f34ef4bc58797f6cf801ff6da" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a"
I20240502 01:55:30.878024 16885 raft_consensus.cc:3037] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.879503 18138 tablet_bootstrap.cc:492] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304: No bootstrap required, opened a new log
I20240502 01:55:30.879662 18138 ts_tablet_manager.cc:1367] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240502 01:55:30.882149 18130 tablet_bootstrap.cc:492] T 53e266d2e3b446c3a50ed3b93a6a5dde P ed2e7e4188d0400b8689e28bbf5096ad: No bootstrap required, opened a new log
I20240502 01:55:30.882359 18130 ts_tablet_manager.cc:1367] T 53e266d2e3b446c3a50ed3b93a6a5dde P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.032s
I20240502 01:55:30.883420 18130 raft_consensus.cc:348] T 53e266d2e3b446c3a50ed3b93a6a5dde P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.883757 18130 raft_consensus.cc:374] T 53e266d2e3b446c3a50ed3b93a6a5dde P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.883906 18130 raft_consensus.cc:729] T 53e266d2e3b446c3a50ed3b93a6a5dde P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.884184 18130 consensus_queue.cc:260] T 53e266d2e3b446c3a50ed3b93a6a5dde P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.884917 18130 ts_tablet_manager.cc:1398] T 53e266d2e3b446c3a50ed3b93a6a5dde P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:55:30.886052 18133 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e: No bootstrap required, opened a new log
I20240502 01:55:30.886081 18138 raft_consensus.cc:348] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.886322 18133 ts_tablet_manager.cc:1367] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e: Time spent bootstrapping tablet: real 0.045s user 0.021s sys 0.016s
I20240502 01:55:30.886361 18138 raft_consensus.cc:374] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.886474 18138 raft_consensus.cc:729] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.887058 16885 raft_consensus.cc:2454] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:30.887210 18133 raft_consensus.cc:348] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:30.887481 18133 raft_consensus.cc:374] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.887547 18133 raft_consensus.cc:729] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4d50250184223992325c86cb6c68e, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.887521 16424 leader_election.cc:302] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 6b9f9aaa1c86420190af75270914449a, dc9fcb8d12cb4ab28ec821c9473adeeb, ed2e7e4188d0400b8689e28bbf5096ad; no voters: 703f6f043c6041a4b629933690580367, cdd4d50250184223992325c86cb6c68e
I20240502 01:55:30.887874 18029 raft_consensus.cc:2781] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:30.887818 18133 consensus_queue.cc:260] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:30.888021 18029 raft_consensus.cc:686] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Becoming Leader. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Running, Role: LEADER
I20240502 01:55:30.888517 18133 ts_tablet_manager.cc:1398] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:55:30.888790 18029 consensus_queue.cc:237] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:30.888887 18138 consensus_queue.cc:260] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.890229 18133 tablet_bootstrap.cc:492] T 53e266d2e3b446c3a50ed3b93a6a5dde P cdd4d50250184223992325c86cb6c68e: Bootstrap starting.
I20240502 01:55:30.890529 18138 ts_tablet_manager.cc:1398] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240502 01:55:30.891849 18133 tablet_bootstrap.cc:654] T 53e266d2e3b446c3a50ed3b93a6a5dde P cdd4d50250184223992325c86cb6c68e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.892143 16753 raft_consensus.cc:2454] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
I20240502 01:55:30.893941 18117 leader_election.cc:288] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 election: Requested vote from peers dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:30.894511 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa6224f34ef4bc58797f6cf801ff6da" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9"
I20240502 01:55:30.895079 17015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cde24f949334006aa9e241e844e81eb" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9"
W20240502 01:55:30.895162 16423 leader_election.cc:341] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:30.895264 17015 raft_consensus.cc:3037] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.895793 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cde24f949334006aa9e241e844e81eb" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a"
I20240502 01:55:30.896024 16885 raft_consensus.cc:3037] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.897367 17015 raft_consensus.cc:2454] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
I20240502 01:55:30.897336 16224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cde24f949334006aa9e241e844e81eb" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e"
I20240502 01:55:30.897547 16224 raft_consensus.cc:3037] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.898839 18044 leader_election.cc:288] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 election: Requested vote from peers ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:30.900000 16224 raft_consensus.cc:2454] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
I20240502 01:55:30.900417 16885 raft_consensus.cc:2454] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
I20240502 01:55:30.900408 16557 leader_election.cc:302] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 703f6f043c6041a4b629933690580367, a6575ebf978a4631a13ba5c243c3c4f9, cdd4d50250184223992325c86cb6c68e, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:30.901036 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cde24f949334006aa9e241e844e81eb" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304"
I20240502 01:55:30.901108 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cde24f949334006aa9e241e844e81eb" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
I20240502 01:55:30.901269 16490 raft_consensus.cc:3037] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Advancing to term 1
W20240502 01:55:30.901715 16558 leader_election.cc:341] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:30.903386 18044 raft_consensus.cc:2781] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:30.903831 18044 raft_consensus.cc:686] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Becoming Leader. State: Replica: 703f6f043c6041a4b629933690580367, State: Running, Role: LEADER
I20240502 01:55:30.903960 16490 raft_consensus.cc:2454] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
I20240502 01:55:30.904214 18044 consensus_queue.cc:237] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.902894 16077 catalog_manager.cc:5484] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39059854b64045698ded1ba4e8d47304 (127.31.161.195). New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:30.906318 18132 ts_tablet_manager.cc:1398] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.082s user 0.002s sys 0.000s
I20240502 01:55:30.906721 18132 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:30.907141 18133 tablet_bootstrap.cc:492] T 53e266d2e3b446c3a50ed3b93a6a5dde P cdd4d50250184223992325c86cb6c68e: No bootstrap required, opened a new log
I20240502 01:55:30.907294 18133 ts_tablet_manager.cc:1367] T 53e266d2e3b446c3a50ed3b93a6a5dde P cdd4d50250184223992325c86cb6c68e: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.006s
I20240502 01:55:30.908066 18133 raft_consensus.cc:348] T 53e266d2e3b446c3a50ed3b93a6a5dde P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.908327 18133 raft_consensus.cc:374] T 53e266d2e3b446c3a50ed3b93a6a5dde P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.908434 18133 raft_consensus.cc:729] T 53e266d2e3b446c3a50ed3b93a6a5dde P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4d50250184223992325c86cb6c68e, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.908703 18133 consensus_queue.cc:260] T 53e266d2e3b446c3a50ed3b93a6a5dde P cdd4d50250184223992325c86cb6c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.909365 18133 ts_tablet_manager.cc:1398] T 53e266d2e3b446c3a50ed3b93a6a5dde P cdd4d50250184223992325c86cb6c68e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:30.911253 18132 tablet_bootstrap.cc:654] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.911346 18135 ts_tablet_manager.cc:1398] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent starting tablet: real 0.093s user 0.010s sys 0.011s
I20240502 01:55:30.912961 18135 tablet_bootstrap.cc:492] T 53e266d2e3b446c3a50ed3b93a6a5dde P dc9fcb8d12cb4ab28ec821c9473adeeb: Bootstrap starting.
I20240502 01:55:30.914634 18135 tablet_bootstrap.cc:654] T 53e266d2e3b446c3a50ed3b93a6a5dde P dc9fcb8d12cb4ab28ec821c9473adeeb: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.917583 18129 tablet_bootstrap.cc:492] T 53e266d2e3b446c3a50ed3b93a6a5dde P 6b9f9aaa1c86420190af75270914449a: No bootstrap required, opened a new log
I20240502 01:55:30.917794 18129 ts_tablet_manager.cc:1367] T 53e266d2e3b446c3a50ed3b93a6a5dde P 6b9f9aaa1c86420190af75270914449a: Time spent bootstrapping tablet: real 0.087s user 0.012s sys 0.011s
I20240502 01:55:30.918606 18129 raft_consensus.cc:348] T 53e266d2e3b446c3a50ed3b93a6a5dde P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.918949 18129 raft_consensus.cc:374] T 53e266d2e3b446c3a50ed3b93a6a5dde P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.918982 18131 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9: No bootstrap required, opened a new log
I20240502 01:55:30.919106 18129 raft_consensus.cc:729] T 53e266d2e3b446c3a50ed3b93a6a5dde P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.919147 18131 ts_tablet_manager.cc:1367] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9: Time spent bootstrapping tablet: real 0.095s user 0.003s sys 0.000s
I20240502 01:55:30.918697 16077 catalog_manager.cc:5484] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 reported cstate change: term changed from 0 to 1, leader changed from <none> to 703f6f043c6041a4b629933690580367 (127.31.161.196). New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:30.919440 18129 consensus_queue.cc:260] T 53e266d2e3b446c3a50ed3b93a6a5dde P 6b9f9aaa1c86420190af75270914449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.919885 18131 raft_consensus.cc:348] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:30.920171 18131 raft_consensus.cc:374] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.920267 18131 raft_consensus.cc:729] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.920373 18129 ts_tablet_manager.cc:1398] T 53e266d2e3b446c3a50ed3b93a6a5dde P 6b9f9aaa1c86420190af75270914449a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:55:30.920518 18131 consensus_queue.cc:260] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:30.921241 18131 ts_tablet_manager.cc:1398] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:55:30.921613 18131 tablet_bootstrap.cc:492] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9: Bootstrap starting.
I20240502 01:55:30.923139 18131 tablet_bootstrap.cc:654] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.924798 18132 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367: No bootstrap required, opened a new log
I20240502 01:55:30.924983 18132 ts_tablet_manager.cc:1367] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.001s
I20240502 01:55:30.925787 18132 raft_consensus.cc:348] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:30.926071 18132 raft_consensus.cc:374] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.926199 18132 raft_consensus.cc:729] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.926471 18132 consensus_queue.cc:260] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:30.927107 18132 ts_tablet_manager.cc:1398] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:30.930611 18131 tablet_bootstrap.cc:492] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9: No bootstrap required, opened a new log
I20240502 01:55:30.930743 18131 ts_tablet_manager.cc:1367] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20240502 01:55:30.931288 18131 raft_consensus.cc:348] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.931552 18131 raft_consensus.cc:374] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.931654 18131 raft_consensus.cc:729] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.931841 18131 consensus_queue.cc:260] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.932334 18131 ts_tablet_manager.cc:1398] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:55:30.937538 18135 tablet_bootstrap.cc:492] T 53e266d2e3b446c3a50ed3b93a6a5dde P dc9fcb8d12cb4ab28ec821c9473adeeb: No bootstrap required, opened a new log
I20240502 01:55:30.937728 18135 ts_tablet_manager.cc:1367] T 53e266d2e3b446c3a50ed3b93a6a5dde P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240502 01:55:30.938386 18135 raft_consensus.cc:348] T 53e266d2e3b446c3a50ed3b93a6a5dde P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.938617 18135 raft_consensus.cc:374] T 53e266d2e3b446c3a50ed3b93a6a5dde P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.938688 18135 raft_consensus.cc:729] T 53e266d2e3b446c3a50ed3b93a6a5dde P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fcb8d12cb4ab28ec821c9473adeeb, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.938922 18135 consensus_queue.cc:260] T 53e266d2e3b446c3a50ed3b93a6a5dde P dc9fcb8d12cb4ab28ec821c9473adeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.939445 18135 ts_tablet_manager.cc:1398] T 53e266d2e3b446c3a50ed3b93a6a5dde P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:55:30.939754 18135 tablet_bootstrap.cc:492] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb: Bootstrap starting.
I20240502 01:55:30.940876 18135 tablet_bootstrap.cc:654] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:30.942684 18135 tablet_bootstrap.cc:492] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb: No bootstrap required, opened a new log
I20240502 01:55:30.942811 18135 ts_tablet_manager.cc:1367] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:55:30.943390 18135 raft_consensus.cc:348] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.943601 18135 raft_consensus.cc:374] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:30.943688 18135 raft_consensus.cc:729] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fcb8d12cb4ab28ec821c9473adeeb, State: Initialized, Role: FOLLOWER
I20240502 01:55:30.943889 18135 consensus_queue.cc:260] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.944341 18135 ts_tablet_manager.cc:1398] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 01:55:30.978757 18046 raft_consensus.cc:482] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:30.978952 18046 raft_consensus.cc:504] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.980648 18046 leader_election.cc:288] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:30.980800 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e266d2e3b446c3a50ed3b93a6a5dde" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:30.981048 16619 raft_consensus.cc:2454] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:30.981607 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e266d2e3b446c3a50ed3b93a6a5dde" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:30.981842 16885 raft_consensus.cc:2454] T 53e266d2e3b446c3a50ed3b93a6a5dde P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:30.981796 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e266d2e3b446c3a50ed3b93a6a5dde" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:30.981827 16224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e266d2e3b446c3a50ed3b93a6a5dde" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:30.982007 16490 raft_consensus.cc:2454] T 53e266d2e3b446c3a50ed3b93a6a5dde P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:30.982043 16224 raft_consensus.cc:2454] T 53e266d2e3b446c3a50ed3b93a6a5dde P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:30.982672 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e266d2e3b446c3a50ed3b93a6a5dde" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:30.982810 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e266d2e3b446c3a50ed3b93a6a5dde" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:30.982894 16359 raft_consensus.cc:2454] T 53e266d2e3b446c3a50ed3b93a6a5dde P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:30.983027 16753 raft_consensus.cc:2454] T 53e266d2e3b446c3a50ed3b93a6a5dde P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:30.983331 16949 leader_election.cc:302] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 6b9f9aaa1c86420190af75270914449a, a6575ebf978a4631a13ba5c243c3c4f9, cdd4d50250184223992325c86cb6c68e; no voters:
I20240502 01:55:30.983608 18046 raft_consensus.cc:2781] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:30.983744 18046 raft_consensus.cc:482] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:30.983839 18046 raft_consensus.cc:3037] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.987160 18046 raft_consensus.cc:504] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.988349 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e266d2e3b446c3a50ed3b93a6a5dde" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367"
I20240502 01:55:30.988590 16619 raft_consensus.cc:3037] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.988871 16224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e266d2e3b446c3a50ed3b93a6a5dde" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e"
I20240502 01:55:30.989050 16224 raft_consensus.cc:3037] T 53e266d2e3b446c3a50ed3b93a6a5dde P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.989372 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e266d2e3b446c3a50ed3b93a6a5dde" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad"
I20240502 01:55:30.989550 16753 raft_consensus.cc:3037] T 53e266d2e3b446c3a50ed3b93a6a5dde P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.991763 16753 raft_consensus.cc:2454] T 53e266d2e3b446c3a50ed3b93a6a5dde P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:30.991899 16224 raft_consensus.cc:2454] T 53e266d2e3b446c3a50ed3b93a6a5dde P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:30.994087 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e266d2e3b446c3a50ed3b93a6a5dde" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a"
I20240502 01:55:30.994313 16885 raft_consensus.cc:3037] T 53e266d2e3b446c3a50ed3b93a6a5dde P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.994468 16619 raft_consensus.cc:2454] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:30.995072 16952 leader_election.cc:302] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 703f6f043c6041a4b629933690580367, a6575ebf978a4631a13ba5c243c3c4f9, cdd4d50250184223992325c86cb6c68e, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:30.995306 18046 leader_election.cc:288] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [CANDIDATE]: Term 1 election: Requested vote from peers 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:30.995486 18046 raft_consensus.cc:2781] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:30.995733 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e266d2e3b446c3a50ed3b93a6a5dde" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
I20240502 01:55:30.995925 18046 raft_consensus.cc:686] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Becoming Leader. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Running, Role: LEADER
I20240502 01:55:30.996039 16359 raft_consensus.cc:3037] T 53e266d2e3b446c3a50ed3b93a6a5dde P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.996268 18046 consensus_queue.cc:237] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:30.996989 16885 raft_consensus.cc:2454] T 53e266d2e3b446c3a50ed3b93a6a5dde P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:30.997368 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53e266d2e3b446c3a50ed3b93a6a5dde" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304"
I20240502 01:55:30.997574 16490 raft_consensus.cc:3037] T 53e266d2e3b446c3a50ed3b93a6a5dde P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:30.998925 16359 raft_consensus.cc:2454] T 53e266d2e3b446c3a50ed3b93a6a5dde P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:30.999935 16490 raft_consensus.cc:2454] T 53e266d2e3b446c3a50ed3b93a6a5dde P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:31.003509 16079 catalog_manager.cc:5484] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199). New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:31.023610 18044 raft_consensus.cc:482] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:31.023819 18044 raft_consensus.cc:504] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:31.025004 18044 leader_election.cc:288] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:31.025377 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa6224f34ef4bc58797f6cf801ff6da" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:31.025718 16753 raft_consensus.cc:2379] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 703f6f043c6041a4b629933690580367 in current term 1: Already voted for candidate 39059854b64045698ded1ba4e8d47304 in this term.
I20240502 01:55:31.025655 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa6224f34ef4bc58797f6cf801ff6da" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:31.025913 16359 raft_consensus.cc:2379] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 703f6f043c6041a4b629933690580367 in current term 1: Already voted for candidate 39059854b64045698ded1ba4e8d47304 in this term.
I20240502 01:55:31.026430 16224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa6224f34ef4bc58797f6cf801ff6da" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:31.026551 17015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa6224f34ef4bc58797f6cf801ff6da" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:31.026687 16224 raft_consensus.cc:2454] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:31.026803 17015 raft_consensus.cc:2454] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:31.027148 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa6224f34ef4bc58797f6cf801ff6da" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:31.027133 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa6224f34ef4bc58797f6cf801ff6da" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:31.027379 16885 raft_consensus.cc:2379] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 703f6f043c6041a4b629933690580367 in current term 1: Already voted for candidate 39059854b64045698ded1ba4e8d47304 in this term.
I20240502 01:55:31.027801 16556 leader_election.cc:302] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 703f6f043c6041a4b629933690580367, a6575ebf978a4631a13ba5c243c3c4f9, cdd4d50250184223992325c86cb6c68e; no voters: 39059854b64045698ded1ba4e8d47304, 6b9f9aaa1c86420190af75270914449a, dc9fcb8d12cb4ab28ec821c9473adeeb, ed2e7e4188d0400b8689e28bbf5096ad
I20240502 01:55:31.028100 18044 raft_consensus.cc:3037] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:31.030133 18044 raft_consensus.cc:2726] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:55:31.033048 18147 raft_consensus.cc:482] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:31.033246 18147 raft_consensus.cc:504] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:31.035534 18147 leader_election.cc:288] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:31.080688 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cde24f949334006aa9e241e844e81eb" candidate_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:31.080953 16885 raft_consensus.cc:2379] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dc9fcb8d12cb4ab28ec821c9473adeeb in current term 1: Already voted for candidate 703f6f043c6041a4b629933690580367 in this term.
I20240502 01:55:31.083854 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cde24f949334006aa9e241e844e81eb" candidate_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:31.090351 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cde24f949334006aa9e241e844e81eb" candidate_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:31.090608 16753 raft_consensus.cc:2379] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dc9fcb8d12cb4ab28ec821c9473adeeb in current term 1: Already voted for candidate 703f6f043c6041a4b629933690580367 in this term.
I20240502 01:55:31.094259 16224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cde24f949334006aa9e241e844e81eb" candidate_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:31.094559 16224 raft_consensus.cc:2379] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dc9fcb8d12cb4ab28ec821c9473adeeb in current term 1: Already voted for candidate 703f6f043c6041a4b629933690580367 in this term.
I20240502 01:55:31.095335 16293 leader_election.cc:302] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: dc9fcb8d12cb4ab28ec821c9473adeeb; no voters: 6b9f9aaa1c86420190af75270914449a, 703f6f043c6041a4b629933690580367, cdd4d50250184223992325c86cb6c68e, ed2e7e4188d0400b8689e28bbf5096ad
I20240502 01:55:31.096005 18147 raft_consensus.cc:3037] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:31.097554 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cde24f949334006aa9e241e844e81eb" candidate_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:31.097808 16490 raft_consensus.cc:2379] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dc9fcb8d12cb4ab28ec821c9473adeeb in current term 1: Already voted for candidate 703f6f043c6041a4b629933690580367 in this term.
I20240502 01:55:31.098847 18147 raft_consensus.cc:2726] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:55:31.100075 17015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cde24f949334006aa9e241e844e81eb" candidate_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:31.100323 17015 raft_consensus.cc:2379] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dc9fcb8d12cb4ab28ec821c9473adeeb in current term 1: Already voted for candidate 703f6f043c6041a4b629933690580367 in this term.
I20240502 01:55:31.294050 18117 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:31.348781 17015 raft_consensus.cc:3037] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:31.352291 18117 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:31.403707 18163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:31.404059 18163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:55:31.404657 18044 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:31.412954 18163 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:55:31.413396 18046 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:31.416393 18117 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:31.427340 18105 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:31.433251 18123 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:31.438030 16224 raft_consensus.cc:3037] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:31.441514 18117 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:31.459260 18044 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:31.462517 18044 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:31.468814 18044 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:31.477773 18046 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:31.484097 18196 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:31.568329 18046 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:31.592797 18123 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:31.610821 18123 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:31.690923 16555 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.199:46837, user_credentials={real_user=slave}} blocked reactor thread for 82941.6us
Using table 'rebalance_test_table_4'
I20240502 01:55:31.714370 18117 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:31.721743 18212 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 371.705 ms
time per row: 0.185852 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 01:55:33.128232 16079 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58366:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 01:55:33.128969 16079 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_5 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240502 01:55:33.138088 16863 tablet_service.cc:1467] Processing CreateTablet for tablet 453dff7b204a431592ac15c058db0493 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.138664 16863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 453dff7b204a431592ac15c058db0493. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.138625 16602 tablet_service.cc:1467] Processing CreateTablet for tablet 453dff7b204a431592ac15c058db0493 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.139091 16602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 453dff7b204a431592ac15c058db0493. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.140808 16864 tablet_service.cc:1467] Processing CreateTablet for tablet a8f7e3c8b6d6434db2a99d7139599a5d (DEFAULT_TABLE table=rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.141260 16864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8f7e3c8b6d6434db2a99d7139599a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.141548 16733 tablet_service.cc:1467] Processing CreateTablet for tablet 453dff7b204a431592ac15c058db0493 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.141968 16733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 453dff7b204a431592ac15c058db0493. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.145572 16204 tablet_service.cc:1467] Processing CreateTablet for tablet 453dff7b204a431592ac15c058db0493 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.146031 16204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 453dff7b204a431592ac15c058db0493. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.146140 16600 tablet_service.cc:1467] Processing CreateTablet for tablet a8f7e3c8b6d6434db2a99d7139599a5d (DEFAULT_TABLE table=rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.146145 16339 tablet_service.cc:1467] Processing CreateTablet for tablet 453dff7b204a431592ac15c058db0493 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.146507 16339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 453dff7b204a431592ac15c058db0493. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.146514 16600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8f7e3c8b6d6434db2a99d7139599a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.148909 16337 tablet_service.cc:1467] Processing CreateTablet for tablet a8f7e3c8b6d6434db2a99d7139599a5d (DEFAULT_TABLE table=rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.149338 16337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8f7e3c8b6d6434db2a99d7139599a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.149451 16205 tablet_service.cc:1467] Processing CreateTablet for tablet a8f7e3c8b6d6434db2a99d7139599a5d (DEFAULT_TABLE table=rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.149820 16205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8f7e3c8b6d6434db2a99d7139599a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.155355 16996 tablet_service.cc:1467] Processing CreateTablet for tablet 453dff7b204a431592ac15c058db0493 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.155566 16469 tablet_service.cc:1467] Processing CreateTablet for tablet 453dff7b204a431592ac15c058db0493 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.155838 16996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 453dff7b204a431592ac15c058db0493. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.155998 16469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 453dff7b204a431592ac15c058db0493. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.159862 16470 tablet_service.cc:1467] Processing CreateTablet for tablet a8f7e3c8b6d6434db2a99d7139599a5d (DEFAULT_TABLE table=rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.160252 16470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8f7e3c8b6d6434db2a99d7139599a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.160912 16732 tablet_service.cc:1467] Processing CreateTablet for tablet a8f7e3c8b6d6434db2a99d7139599a5d (DEFAULT_TABLE table=rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.161408 16732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8f7e3c8b6d6434db2a99d7139599a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.174291 18299 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e: Bootstrap starting.
I20240502 01:55:33.174430 16995 tablet_service.cc:1467] Processing CreateTablet for tablet a8f7e3c8b6d6434db2a99d7139599a5d (DEFAULT_TABLE table=rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.174914 16995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8f7e3c8b6d6434db2a99d7139599a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.182773 18299 tablet_bootstrap.cc:654] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.184474 18295 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:33.188097 16338 tablet_service.cc:1467] Processing CreateTablet for tablet 27ce7f50b71e4f478cace3bde6f5e1a3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.188364 16471 tablet_service.cc:1467] Processing CreateTablet for tablet 27ce7f50b71e4f478cace3bde6f5e1a3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.188618 16338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27ce7f50b71e4f478cace3bde6f5e1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.188831 16471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27ce7f50b71e4f478cace3bde6f5e1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.189855 16863 tablet_service.cc:1467] Processing CreateTablet for tablet 27ce7f50b71e4f478cace3bde6f5e1a3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.190248 16863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27ce7f50b71e4f478cace3bde6f5e1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.190263 16994 tablet_service.cc:1467] Processing CreateTablet for tablet 27ce7f50b71e4f478cace3bde6f5e1a3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.190652 16994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27ce7f50b71e4f478cace3bde6f5e1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.195749 16733 tablet_service.cc:1467] Processing CreateTablet for tablet 27ce7f50b71e4f478cace3bde6f5e1a3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.196271 16733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27ce7f50b71e4f478cace3bde6f5e1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.201645 16204 tablet_service.cc:1467] Processing CreateTablet for tablet 27ce7f50b71e4f478cace3bde6f5e1a3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.201889 18295 tablet_bootstrap.cc:654] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.202198 16204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27ce7f50b71e4f478cace3bde6f5e1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.201961 16602 tablet_service.cc:1467] Processing CreateTablet for tablet 27ce7f50b71e4f478cace3bde6f5e1a3 (DEFAULT_TABLE table=rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:33.203758 16602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27ce7f50b71e4f478cace3bde6f5e1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:33.207648 18301 tablet_bootstrap.cc:492] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304: Bootstrap starting.
I20240502 01:55:33.213450 18299 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e: No bootstrap required, opened a new log
I20240502 01:55:33.213590 18300 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9: Bootstrap starting.
I20240502 01:55:33.215781 18300 tablet_bootstrap.cc:654] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.215845 18297 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a: Bootstrap starting.
I20240502 01:55:33.217278 18299 ts_tablet_manager.cc:1367] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e: Time spent bootstrapping tablet: real 0.043s user 0.001s sys 0.003s
I20240502 01:55:33.218080 18295 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367: No bootstrap required, opened a new log
I20240502 01:55:33.218333 18295 ts_tablet_manager.cc:1367] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.004s
I20240502 01:55:33.218228 18299 raft_consensus.cc:348] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.218585 18301 tablet_bootstrap.cc:654] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.218696 18296 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb: Bootstrap starting.
I20240502 01:55:33.219238 18295 raft_consensus.cc:348] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:33.219574 18295 raft_consensus.cc:374] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.219651 18299 raft_consensus.cc:374] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.219727 18295 raft_consensus.cc:729] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.219831 18299 raft_consensus.cc:729] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4d50250184223992325c86cb6c68e, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.220045 18295 consensus_queue.cc:260] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:33.220417 18296 tablet_bootstrap.cc:654] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.221202 18295 ts_tablet_manager.cc:1398] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:55:33.220384 18299 consensus_queue.cc:260] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.221760 18298 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:33.221941 18295 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:33.223143 18301 tablet_bootstrap.cc:492] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304: No bootstrap required, opened a new log
I20240502 01:55:33.223371 18298 tablet_bootstrap.cc:654] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.223358 18301 ts_tablet_manager.cc:1367] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20240502 01:55:33.223526 18295 tablet_bootstrap.cc:654] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.225654 18301 raft_consensus.cc:348] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.226358 18301 raft_consensus.cc:374] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.227301 18301 raft_consensus.cc:729] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.227674 18301 consensus_queue.cc:260] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.228080 18300 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9: No bootstrap required, opened a new log
I20240502 01:55:33.228296 18300 ts_tablet_manager.cc:1367] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240502 01:55:33.228415 18301 ts_tablet_manager.cc:1398] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 01:55:33.229749 18299 ts_tablet_manager.cc:1398] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240502 01:55:33.230167 18299 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e: Bootstrap starting.
I20240502 01:55:33.231374 18297 tablet_bootstrap.cc:654] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.232002 18299 tablet_bootstrap.cc:654] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.232333 18295 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367: No bootstrap required, opened a new log
I20240502 01:55:33.232565 18295 ts_tablet_manager.cc:1367] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240502 01:55:33.232985 18301 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304: Bootstrap starting.
I20240502 01:55:33.233336 18295 raft_consensus.cc:348] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.233695 18295 raft_consensus.cc:374] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.233824 18295 raft_consensus.cc:729] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.234158 18295 consensus_queue.cc:260] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.234694 18301 tablet_bootstrap.cc:654] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.234957 18295 ts_tablet_manager.cc:1398] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:33.235294 18295 tablet_bootstrap.cc:492] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:33.236356 18300 raft_consensus.cc:348] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.236725 18300 raft_consensus.cc:374] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.236840 18295 tablet_bootstrap.cc:654] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.236871 18300 raft_consensus.cc:729] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.238299 18300 consensus_queue.cc:260] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.239173 18300 ts_tablet_manager.cc:1398] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:55:33.239717 18300 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9: Bootstrap starting.
I20240502 01:55:33.240517 18295 tablet_bootstrap.cc:492] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367: No bootstrap required, opened a new log
I20240502 01:55:33.240722 18295 ts_tablet_manager.cc:1367] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 01:55:33.241315 18300 tablet_bootstrap.cc:654] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.243058 18295 raft_consensus.cc:348] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.243273 18295 raft_consensus.cc:374] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.243364 18295 raft_consensus.cc:729] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.243588 18295 consensus_queue.cc:260] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.244297 18295 ts_tablet_manager.cc:1398] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:55:33.248440 18298 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad: No bootstrap required, opened a new log
I20240502 01:55:33.248839 18298 ts_tablet_manager.cc:1367] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20240502 01:55:33.249181 18296 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb: No bootstrap required, opened a new log
I20240502 01:55:33.249409 18296 ts_tablet_manager.cc:1367] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20240502 01:55:33.249583 18301 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304: No bootstrap required, opened a new log
I20240502 01:55:33.249794 18301 ts_tablet_manager.cc:1367] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.002s
I20240502 01:55:33.249722 18298 raft_consensus.cc:348] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.250001 18298 raft_consensus.cc:374] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.250097 18298 raft_consensus.cc:729] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.250370 18298 consensus_queue.cc:260] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.250526 18296 raft_consensus.cc:348] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.250996 18298 ts_tablet_manager.cc:1398] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:55:33.251034 18296 raft_consensus.cc:374] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.251209 18296 raft_consensus.cc:729] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fcb8d12cb4ab28ec821c9473adeeb, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.251580 18298 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:33.253312 18298 tablet_bootstrap.cc:654] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.251509 18296 consensus_queue.cc:260] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.254030 18299 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e: No bootstrap required, opened a new log
I20240502 01:55:33.254231 18300 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9: No bootstrap required, opened a new log
I20240502 01:55:33.254271 18299 ts_tablet_manager.cc:1367] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.001s
I20240502 01:55:33.254437 18300 ts_tablet_manager.cc:1367] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240502 01:55:33.255141 18297 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a: No bootstrap required, opened a new log
I20240502 01:55:33.255049 18299 raft_consensus.cc:348] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:33.255323 18297 ts_tablet_manager.cc:1367] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.003s
I20240502 01:55:33.255345 18299 raft_consensus.cc:374] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.255424 18299 raft_consensus.cc:729] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4d50250184223992325c86cb6c68e, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.255198 18300 raft_consensus.cc:348] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:33.255776 18300 raft_consensus.cc:374] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.255710 18299 consensus_queue.cc:260] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:33.255914 18300 raft_consensus.cc:729] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.256132 18297 raft_consensus.cc:348] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.256443 18297 raft_consensus.cc:374] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.256546 18297 raft_consensus.cc:729] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.256925 18299 ts_tablet_manager.cc:1398] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:55:33.256801 18297 consensus_queue.cc:260] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.257452 18297 ts_tablet_manager.cc:1398] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 01:55:33.258350 18299 tablet_bootstrap.cc:492] T 27ce7f50b71e4f478cace3bde6f5e1a3 P cdd4d50250184223992325c86cb6c68e: Bootstrap starting.
I20240502 01:55:33.258756 18297 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a: Bootstrap starting.
I20240502 01:55:33.256161 18300 consensus_queue.cc:260] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:33.250541 18301 raft_consensus.cc:348] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.260854 18301 raft_consensus.cc:374] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.260982 18300 ts_tablet_manager.cc:1398] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:55:33.261344 18300 tablet_bootstrap.cc:492] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a6575ebf978a4631a13ba5c243c3c4f9: Bootstrap starting.
I20240502 01:55:33.261865 18301 raft_consensus.cc:729] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.262215 18301 consensus_queue.cc:260] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.263557 18297 tablet_bootstrap.cc:654] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.264072 18301 ts_tablet_manager.cc:1398] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240502 01:55:33.264638 18299 tablet_bootstrap.cc:654] T 27ce7f50b71e4f478cace3bde6f5e1a3 P cdd4d50250184223992325c86cb6c68e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.264827 18300 tablet_bootstrap.cc:654] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a6575ebf978a4631a13ba5c243c3c4f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.265007 18301 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304: Bootstrap starting.
I20240502 01:55:33.268112 18301 tablet_bootstrap.cc:654] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.269641 18289 raft_consensus.cc:482] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:33.269824 18289 raft_consensus.cc:504] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.271049 18289 leader_election.cc:288] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121)
I20240502 01:55:33.272063 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f7e3c8b6d6434db2a99d7139599a5d" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:33.272338 16753 raft_consensus.cc:2454] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:33.273053 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f7e3c8b6d6434db2a99d7139599a5d" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:33.273315 16490 raft_consensus.cc:2454] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:33.274250 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f7e3c8b6d6434db2a99d7139599a5d" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:33.274480 16885 raft_consensus.cc:2454] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:33.275862 16950 leader_election.cc:302] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 6b9f9aaa1c86420190af75270914449a, a6575ebf978a4631a13ba5c243c3c4f9, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:33.276762 18299 tablet_bootstrap.cc:492] T 27ce7f50b71e4f478cace3bde6f5e1a3 P cdd4d50250184223992325c86cb6c68e: No bootstrap required, opened a new log
I20240502 01:55:33.277274 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f7e3c8b6d6434db2a99d7139599a5d" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:33.277551 16359 raft_consensus.cc:2454] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:33.277866 18299 ts_tablet_manager.cc:1367] T 27ce7f50b71e4f478cace3bde6f5e1a3 P cdd4d50250184223992325c86cb6c68e: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.006s
I20240502 01:55:33.278759 18297 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a: No bootstrap required, opened a new log
I20240502 01:55:33.279738 18296 ts_tablet_manager.cc:1398] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent starting tablet: real 0.030s user 0.012s sys 0.013s
I20240502 01:55:33.279889 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f7e3c8b6d6434db2a99d7139599a5d" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:33.280134 16619 raft_consensus.cc:2454] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:33.280187 18296 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb: Bootstrap starting.
I20240502 01:55:33.280580 18297 ts_tablet_manager.cc:1367] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.005s
I20240502 01:55:33.281386 18297 raft_consensus.cc:348] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:33.281806 18296 tablet_bootstrap.cc:654] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.283035 18299 raft_consensus.cc:348] T 27ce7f50b71e4f478cace3bde6f5e1a3 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.283488 18299 raft_consensus.cc:374] T 27ce7f50b71e4f478cace3bde6f5e1a3 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.283648 18299 raft_consensus.cc:729] T 27ce7f50b71e4f478cace3bde6f5e1a3 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4d50250184223992325c86cb6c68e, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.283686 16224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f7e3c8b6d6434db2a99d7139599a5d" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:33.283922 16224 raft_consensus.cc:2454] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 0.
I20240502 01:55:33.283979 18300 tablet_bootstrap.cc:492] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a6575ebf978a4631a13ba5c243c3c4f9: No bootstrap required, opened a new log
I20240502 01:55:33.284132 18300 ts_tablet_manager.cc:1367] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.008s
I20240502 01:55:33.284123 18297 raft_consensus.cc:374] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.284200 18297 raft_consensus.cc:729] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.284315 18299 consensus_queue.cc:260] T 27ce7f50b71e4f478cace3bde6f5e1a3 P cdd4d50250184223992325c86cb6c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.285171 18300 raft_consensus.cc:348] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.285477 18300 raft_consensus.cc:374] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.285622 18300 raft_consensus.cc:729] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.285912 18300 consensus_queue.cc:260] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a6575ebf978a4631a13ba5c243c3c4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.286455 18218 raft_consensus.cc:2781] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:33.286541 18218 raft_consensus.cc:482] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:33.286638 18218 raft_consensus.cc:3037] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.286775 18299 ts_tablet_manager.cc:1398] T 27ce7f50b71e4f478cace3bde6f5e1a3 P cdd4d50250184223992325c86cb6c68e: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 01:55:33.284857 18297 consensus_queue.cc:260] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:33.287535 18297 ts_tablet_manager.cc:1398] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 01:55:33.287823 18297 tablet_bootstrap.cc:492] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 6b9f9aaa1c86420190af75270914449a: Bootstrap starting.
I20240502 01:55:33.289894 18218 raft_consensus.cc:504] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.290108 18296 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb: No bootstrap required, opened a new log
I20240502 01:55:33.290115 18301 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304: No bootstrap required, opened a new log
I20240502 01:55:33.290256 18301 ts_tablet_manager.cc:1367] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.004s
I20240502 01:55:33.290274 18296 ts_tablet_manager.cc:1367] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240502 01:55:33.291034 18296 raft_consensus.cc:348] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:33.291180 16224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f7e3c8b6d6434db2a99d7139599a5d" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e"
I20240502 01:55:33.291373 16224 raft_consensus.cc:3037] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.291253 18301 raft_consensus.cc:348] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:33.291414 18296 raft_consensus.cc:374] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.291564 18296 raft_consensus.cc:729] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fcb8d12cb4ab28ec821c9473adeeb, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.291581 18301 raft_consensus.cc:374] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.291695 18301 raft_consensus.cc:729] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.291877 18296 consensus_queue.cc:260] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:33.292003 18301 consensus_queue.cc:260] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:33.292448 18296 ts_tablet_manager.cc:1398] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:55:33.292804 18296 tablet_bootstrap.cc:492] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb: Bootstrap starting.
I20240502 01:55:33.293545 18301 ts_tablet_manager.cc:1398] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 01:55:33.294078 16224 raft_consensus.cc:2454] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:33.294557 18296 tablet_bootstrap.cc:654] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.294612 18297 tablet_bootstrap.cc:654] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 6b9f9aaa1c86420190af75270914449a: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.295228 16358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f7e3c8b6d6434db2a99d7139599a5d" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
I20240502 01:55:33.295447 16358 raft_consensus.cc:3037] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.298583 16358 raft_consensus.cc:2454] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:33.299546 18218 leader_election.cc:288] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [CANDIDATE]: Term 1 election: Requested vote from peers cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121)
I20240502 01:55:33.300051 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f7e3c8b6d6434db2a99d7139599a5d" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a"
I20240502 01:55:33.300470 16885 raft_consensus.cc:3037] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.301530 18300 ts_tablet_manager.cc:1398] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20240502 01:55:33.304530 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f7e3c8b6d6434db2a99d7139599a5d" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367"
I20240502 01:55:33.304819 16619 raft_consensus.cc:3037] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.307891 16619 raft_consensus.cc:2454] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:33.308444 16952 leader_election.cc:302] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 703f6f043c6041a4b629933690580367, a6575ebf978a4631a13ba5c243c3c4f9, cdd4d50250184223992325c86cb6c68e, dc9fcb8d12cb4ab28ec821c9473adeeb; no voters:
I20240502 01:55:33.308521 18287 raft_consensus.cc:482] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:33.308801 18218 raft_consensus.cc:2781] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:33.308693 18287 raft_consensus.cc:504] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:33.308941 18218 raft_consensus.cc:686] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Becoming Leader. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Running, Role: LEADER
I20240502 01:55:33.309309 18218 consensus_queue.cc:237] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.310114 18287 leader_election.cc:288] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:33.311424 16358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453dff7b204a431592ac15c058db0493" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:33.311678 16358 raft_consensus.cc:2454] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 0.
I20240502 01:55:33.312080 16224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453dff7b204a431592ac15c058db0493" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:33.312330 16224 raft_consensus.cc:2454] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 0.
I20240502 01:55:33.313202 17014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453dff7b204a431592ac15c058db0493" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:33.313452 17014 raft_consensus.cc:2454] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 0.
I20240502 01:55:33.314814 16818 leader_election.cc:302] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 6b9f9aaa1c86420190af75270914449a, a6575ebf978a4631a13ba5c243c3c4f9, cdd4d50250184223992325c86cb6c68e, dc9fcb8d12cb4ab28ec821c9473adeeb; no voters:
I20240502 01:55:33.315359 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453dff7b204a431592ac15c058db0493" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:33.315626 16490 raft_consensus.cc:2454] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 0.
I20240502 01:55:33.316711 18297 tablet_bootstrap.cc:492] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 6b9f9aaa1c86420190af75270914449a: No bootstrap required, opened a new log
I20240502 01:55:33.317713 18297 ts_tablet_manager.cc:1367] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 6b9f9aaa1c86420190af75270914449a: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.004s
I20240502 01:55:33.318850 18297 raft_consensus.cc:348] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.320271 18297 raft_consensus.cc:374] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.320513 18297 raft_consensus.cc:729] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.320910 18297 consensus_queue.cc:260] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 6b9f9aaa1c86420190af75270914449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.321904 18055 raft_consensus.cc:2781] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:33.321990 18055 raft_consensus.cc:482] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:33.322098 18055 raft_consensus.cc:3037] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.322816 18297 ts_tablet_manager.cc:1398] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 6b9f9aaa1c86420190af75270914449a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:55:33.323072 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453dff7b204a431592ac15c058db0493" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
W20240502 01:55:33.323948 16818 leader_election.cc:341] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:33.343040 18055 raft_consensus.cc:504] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:33.344425 18284 raft_consensus.cc:482] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:33.344619 18284 raft_consensus.cc:504] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.345893 18284 leader_election.cc:288] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:33.347133 16884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ce7f50b71e4f478cace3bde6f5e1a3" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:33.347154 16224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ce7f50b71e4f478cace3bde6f5e1a3" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:33.347357 16884 raft_consensus.cc:2454] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:33.347374 16224 raft_consensus.cc:2454] T 27ce7f50b71e4f478cace3bde6f5e1a3 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:33.349375 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453dff7b204a431592ac15c058db0493" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
I20240502 01:55:33.349494 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453dff7b204a431592ac15c058db0493" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367"
I20240502 01:55:33.349684 16359 raft_consensus.cc:3037] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.349771 16621 raft_consensus.cc:3037] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.352797 16621 raft_consensus.cc:2454] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 1.
I20240502 01:55:33.353006 16359 raft_consensus.cc:2454] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 1.
I20240502 01:55:33.354519 17014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453dff7b204a431592ac15c058db0493" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9"
I20240502 01:55:33.354770 17014 raft_consensus.cc:3037] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.357988 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453dff7b204a431592ac15c058db0493" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad"
W20240502 01:55:33.360306 16818 leader_election.cc:341] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:33.363153 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ce7f50b71e4f478cace3bde6f5e1a3" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
W20240502 01:55:33.363938 16423 leader_election.cc:341] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:33.365658 16357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ce7f50b71e4f478cace3bde6f5e1a3" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
W20240502 01:55:33.367194 16426 leader_election.cc:341] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:33.367623 16621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ce7f50b71e4f478cace3bde6f5e1a3" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:33.367916 16621 raft_consensus.cc:2454] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:33.369210 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453dff7b204a431592ac15c058db0493" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304"
I20240502 01:55:33.369421 16490 raft_consensus.cc:3037] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.374071 16224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453dff7b204a431592ac15c058db0493" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e"
I20240502 01:55:33.374344 16224 raft_consensus.cc:3037] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.381964 17014 raft_consensus.cc:2454] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 1.
I20240502 01:55:33.382280 16885 raft_consensus.cc:2454] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:33.384581 16426 leader_election.cc:302] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 6b9f9aaa1c86420190af75270914449a, 703f6f043c6041a4b629933690580367, cdd4d50250184223992325c86cb6c68e; no voters: dc9fcb8d12cb4ab28ec821c9473adeeb, ed2e7e4188d0400b8689e28bbf5096ad
I20240502 01:55:33.385174 18284 raft_consensus.cc:2781] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:33.385311 18284 raft_consensus.cc:482] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:33.385387 18284 raft_consensus.cc:3037] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.388286 18284 raft_consensus.cc:504] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.389865 16225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ce7f50b71e4f478cace3bde6f5e1a3" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e"
I20240502 01:55:33.390111 16225 raft_consensus.cc:3037] T 27ce7f50b71e4f478cace3bde6f5e1a3 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.393035 16225 raft_consensus.cc:2454] T 27ce7f50b71e4f478cace3bde6f5e1a3 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:33.383960 18055 leader_election.cc:288] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [CANDIDATE]: Term 1 election: Requested vote from peers ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:33.397895 16490 raft_consensus.cc:2454] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 1.
I20240502 01:55:33.398322 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453dff7b204a431592ac15c058db0493" candidate_uuid: "6b9f9aaa1c86420190af75270914449a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:33.398983 16224 raft_consensus.cc:2454] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 1.
I20240502 01:55:33.399772 16818 leader_election.cc:302] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 6b9f9aaa1c86420190af75270914449a, 703f6f043c6041a4b629933690580367, dc9fcb8d12cb4ab28ec821c9473adeeb; no voters: ed2e7e4188d0400b8689e28bbf5096ad
I20240502 01:55:33.400171 18055 raft_consensus.cc:2781] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:33.400441 18055 raft_consensus.cc:686] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [term 1 LEADER]: Becoming Leader. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Running, Role: LEADER
I20240502 01:55:33.400876 18055 consensus_queue.cc:237] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:33.401669 16619 raft_consensus.cc:2362] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6b9f9aaa1c86420190af75270914449a in term 1. Re-sending same reply.
I20240502 01:55:33.408677 16357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ce7f50b71e4f478cace3bde6f5e1a3" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
W20240502 01:55:33.409649 16426 leader_election.cc:341] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:33.412688 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ce7f50b71e4f478cace3bde6f5e1a3" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367"
I20240502 01:55:33.412959 16619 raft_consensus.cc:3037] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.415916 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ce7f50b71e4f478cace3bde6f5e1a3" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a"
I20240502 01:55:33.416229 16885 raft_consensus.cc:3037] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.423321 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ce7f50b71e4f478cace3bde6f5e1a3" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad"
W20240502 01:55:33.424001 16423 leader_election.cc:341] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:33.426980 16082 catalog_manager.cc:5484] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b9f9aaa1c86420190af75270914449a (127.31.161.198). New cstate: current_term: 1 leader_uuid: "6b9f9aaa1c86420190af75270914449a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:33.428367 18284 leader_election.cc:288] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 election: Requested vote from peers cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:33.430852 16885 raft_consensus.cc:2454] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:33.435624 16619 raft_consensus.cc:2454] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:33.438637 16426 leader_election.cc:302] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 6b9f9aaa1c86420190af75270914449a, 703f6f043c6041a4b629933690580367, cdd4d50250184223992325c86cb6c68e; no voters: dc9fcb8d12cb4ab28ec821c9473adeeb, ed2e7e4188d0400b8689e28bbf5096ad
I20240502 01:55:33.438977 18192 raft_consensus.cc:2781] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:33.439134 18192 raft_consensus.cc:686] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Becoming Leader. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Running, Role: LEADER
I20240502 01:55:33.439487 18192 consensus_queue.cc:237] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.441777 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f7e3c8b6d6434db2a99d7139599a5d" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304"
I20240502 01:55:33.442071 16490 raft_consensus.cc:3037] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.445134 16490 raft_consensus.cc:2454] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:33.446499 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f7e3c8b6d6434db2a99d7139599a5d" candidate_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad"
I20240502 01:55:33.446774 16753 raft_consensus.cc:3037] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.449885 16753 raft_consensus.cc:2454] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6575ebf978a4631a13ba5c243c3c4f9 in term 1.
I20240502 01:55:33.452391 17016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ce7f50b71e4f478cace3bde6f5e1a3" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9"
I20240502 01:55:33.452631 17016 raft_consensus.cc:3037] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.455386 17016 raft_consensus.cc:2454] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:33.457569 16082 catalog_manager.cc:5484] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39059854b64045698ded1ba4e8d47304 (127.31.161.195). New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:33.458242 17014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ce7f50b71e4f478cace3bde6f5e1a3" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:33.458506 17014 raft_consensus.cc:2362] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1. Re-sending same reply.
I20240502 01:55:33.465557 18298 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad: No bootstrap required, opened a new log
I20240502 01:55:33.465735 18298 ts_tablet_manager.cc:1367] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.214s user 0.062s sys 0.098s
I20240502 01:55:33.465991 16082 catalog_manager.cc:5484] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199). New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:33.466374 18298 raft_consensus.cc:348] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:33.466631 18298 raft_consensus.cc:374] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.466708 18298 raft_consensus.cc:729] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.466967 18298 consensus_queue.cc:260] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:33.467619 18298 ts_tablet_manager.cc:1398] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:33.467983 18298 tablet_bootstrap.cc:492] T 27ce7f50b71e4f478cace3bde6f5e1a3 P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:33.469552 18298 tablet_bootstrap.cc:654] T 27ce7f50b71e4f478cace3bde6f5e1a3 P ed2e7e4188d0400b8689e28bbf5096ad: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:33.471016 18296 tablet_bootstrap.cc:492] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb: No bootstrap required, opened a new log
I20240502 01:55:33.471218 18296 ts_tablet_manager.cc:1367] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent bootstrapping tablet: real 0.179s user 0.041s sys 0.107s
I20240502 01:55:33.472092 18296 raft_consensus.cc:348] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.472394 18296 raft_consensus.cc:374] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.472505 18296 raft_consensus.cc:729] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fcb8d12cb4ab28ec821c9473adeeb, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.472918 18296 consensus_queue.cc:260] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.474614 18296 ts_tablet_manager.cc:1398] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:55:33.475847 18298 tablet_bootstrap.cc:492] T 27ce7f50b71e4f478cace3bde6f5e1a3 P ed2e7e4188d0400b8689e28bbf5096ad: No bootstrap required, opened a new log
I20240502 01:55:33.476035 18298 ts_tablet_manager.cc:1367] T 27ce7f50b71e4f478cace3bde6f5e1a3 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 01:55:33.476754 18298 raft_consensus.cc:348] T 27ce7f50b71e4f478cace3bde6f5e1a3 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.477035 18298 raft_consensus.cc:374] T 27ce7f50b71e4f478cace3bde6f5e1a3 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:33.477190 18298 raft_consensus.cc:729] T 27ce7f50b71e4f478cace3bde6f5e1a3 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: FOLLOWER
I20240502 01:55:33.477455 18298 consensus_queue.cc:260] T 27ce7f50b71e4f478cace3bde6f5e1a3 P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.478119 18298 ts_tablet_manager.cc:1398] T 27ce7f50b71e4f478cace3bde6f5e1a3 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:33.568307 18286 raft_consensus.cc:482] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:33.568499 18286 raft_consensus.cc:504] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:33.569780 18286 leader_election.cc:288] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:33.569820 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453dff7b204a431592ac15c058db0493" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:33.570271 16357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453dff7b204a431592ac15c058db0493" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:33.570546 16357 raft_consensus.cc:2379] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ed2e7e4188d0400b8689e28bbf5096ad in current term 1: Already voted for candidate 6b9f9aaa1c86420190af75270914449a in this term.
I20240502 01:55:33.571266 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453dff7b204a431592ac15c058db0493" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:33.571336 16224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453dff7b204a431592ac15c058db0493" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:33.571525 16490 raft_consensus.cc:2379] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ed2e7e4188d0400b8689e28bbf5096ad in current term 1: Already voted for candidate 6b9f9aaa1c86420190af75270914449a in this term.
I20240502 01:55:33.571611 16224 raft_consensus.cc:2379] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ed2e7e4188d0400b8689e28bbf5096ad in current term 1: Already voted for candidate 6b9f9aaa1c86420190af75270914449a in this term.
I20240502 01:55:33.572010 17014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453dff7b204a431592ac15c058db0493" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:33.572309 17014 raft_consensus.cc:2379] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ed2e7e4188d0400b8689e28bbf5096ad in current term 1: Already voted for candidate 6b9f9aaa1c86420190af75270914449a in this term.
I20240502 01:55:33.572396 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "453dff7b204a431592ac15c058db0493" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:33.572718 16619 raft_consensus.cc:2379] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ed2e7e4188d0400b8689e28bbf5096ad in current term 1: Already voted for candidate 6b9f9aaa1c86420190af75270914449a in this term.
I20240502 01:55:33.572783 16687 leader_election.cc:302] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: ed2e7e4188d0400b8689e28bbf5096ad; no voters: 6b9f9aaa1c86420190af75270914449a, a6575ebf978a4631a13ba5c243c3c4f9, cdd4d50250184223992325c86cb6c68e, dc9fcb8d12cb4ab28ec821c9473adeeb
I20240502 01:55:33.573261 18286 raft_consensus.cc:3037] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.575606 18286 raft_consensus.cc:2726] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:55:33.700013 18218 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:33.738669 18218 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:33.820588 18055 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:33.835541 18315 raft_consensus.cc:482] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:33.835781 18315 raft_consensus.cc:504] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:33.837275 18315 leader_election.cc:288] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:33.837630 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ce7f50b71e4f478cace3bde6f5e1a3" candidate_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:33.837906 16885 raft_consensus.cc:2379] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dc9fcb8d12cb4ab28ec821c9473adeeb in current term 1: Already voted for candidate 39059854b64045698ded1ba4e8d47304 in this term.
I20240502 01:55:33.837949 17014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ce7f50b71e4f478cace3bde6f5e1a3" candidate_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:33.837981 16753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ce7f50b71e4f478cace3bde6f5e1a3" candidate_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:33.838253 17014 raft_consensus.cc:2379] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dc9fcb8d12cb4ab28ec821c9473adeeb in current term 1: Already voted for candidate 39059854b64045698ded1ba4e8d47304 in this term.
I20240502 01:55:33.838261 16753 raft_consensus.cc:2454] T 27ce7f50b71e4f478cace3bde6f5e1a3 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc9fcb8d12cb4ab28ec821c9473adeeb in term 0.
I20240502 01:55:33.838757 16224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ce7f50b71e4f478cace3bde6f5e1a3" candidate_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:33.839078 16224 raft_consensus.cc:2379] T 27ce7f50b71e4f478cace3bde6f5e1a3 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dc9fcb8d12cb4ab28ec821c9473adeeb in current term 1: Already voted for candidate 39059854b64045698ded1ba4e8d47304 in this term.
I20240502 01:55:33.839411 16491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ce7f50b71e4f478cace3bde6f5e1a3" candidate_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:33.840094 16291 leader_election.cc:302] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: dc9fcb8d12cb4ab28ec821c9473adeeb, ed2e7e4188d0400b8689e28bbf5096ad; no voters: 39059854b64045698ded1ba4e8d47304, 6b9f9aaa1c86420190af75270914449a, a6575ebf978a4631a13ba5c243c3c4f9, cdd4d50250184223992325c86cb6c68e
I20240502 01:55:33.840201 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27ce7f50b71e4f478cace3bde6f5e1a3" candidate_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:33.840498 16619 raft_consensus.cc:2379] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dc9fcb8d12cb4ab28ec821c9473adeeb in current term 1: Already voted for candidate 39059854b64045698ded1ba4e8d47304 in this term.
I20240502 01:55:33.841370 18315 raft_consensus.cc:3037] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.843533 18315 raft_consensus.cc:2726] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 01:55:33.853776 18192 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:33.869354 16753 raft_consensus.cc:3037] T 27ce7f50b71e4f478cace3bde6f5e1a3 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:33.872139 18192 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:33.883829 18055 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:33.885973 18329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:33.886366 18329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:55:33.887784 18218 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:33.894366 18289 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:33.897121 18329 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:55:33.897378 18289 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:33.905951 18192 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:33.916062 18055 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:33.923913 18284 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:33.927019 18284 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:33.929787 18289 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:33.936259 18284 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:33.945179 18055 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:33.974792 18055 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 01:55:34.004954 18368 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:34.016299 16426 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:38133, user_credentials={real_user=slave}} blocked reactor thread for 52548.1us
W20240502 01:55:34.020668 16425 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:41953, user_credentials={real_user=slave}} blocked reactor thread for 46490.4us
Using table 'rebalance_test_table_5'
INSERT report
rows total: 2000
time total: 500.535 ms
time per row: 0.250268 ms
I20240502 01:55:34.675941 16082 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58366:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 01:55:34.676790 16082 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_6 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240502 01:55:34.686764 16204 tablet_service.cc:1467] Processing CreateTablet for tablet 479fcadbde0449f4bfeef0bd2f82ebaf (DEFAULT_TABLE table=rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.686903 16337 tablet_service.cc:1467] Processing CreateTablet for tablet 479fcadbde0449f4bfeef0bd2f82ebaf (DEFAULT_TABLE table=rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.687155 16471 tablet_service.cc:1467] Processing CreateTablet for tablet 479fcadbde0449f4bfeef0bd2f82ebaf (DEFAULT_TABLE table=rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.687335 16204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 479fcadbde0449f4bfeef0bd2f82ebaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.687364 16337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 479fcadbde0449f4bfeef0bd2f82ebaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.687597 16471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 479fcadbde0449f4bfeef0bd2f82ebaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.689527 16864 tablet_service.cc:1467] Processing CreateTablet for tablet 479fcadbde0449f4bfeef0bd2f82ebaf (DEFAULT_TABLE table=rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.689954 16864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 479fcadbde0449f4bfeef0bd2f82ebaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.690511 16863 tablet_service.cc:1467] Processing CreateTablet for tablet bc9cc69a319d46b8940f4c7dc2e7e656 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.690999 16863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9cc69a319d46b8940f4c7dc2e7e656. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.692225 16995 tablet_service.cc:1467] Processing CreateTablet for tablet 479fcadbde0449f4bfeef0bd2f82ebaf (DEFAULT_TABLE table=rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.692695 16995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 479fcadbde0449f4bfeef0bd2f82ebaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.692663 16602 tablet_service.cc:1467] Processing CreateTablet for tablet 479fcadbde0449f4bfeef0bd2f82ebaf (DEFAULT_TABLE table=rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.693058 16602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 479fcadbde0449f4bfeef0bd2f82ebaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.694129 16600 tablet_service.cc:1467] Processing CreateTablet for tablet bc9cc69a319d46b8940f4c7dc2e7e656 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.694584 16600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9cc69a319d46b8940f4c7dc2e7e656. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.697969 16733 tablet_service.cc:1467] Processing CreateTablet for tablet bc9cc69a319d46b8940f4c7dc2e7e656 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.698451 16733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9cc69a319d46b8940f4c7dc2e7e656. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.697964 16732 tablet_service.cc:1467] Processing CreateTablet for tablet 479fcadbde0449f4bfeef0bd2f82ebaf (DEFAULT_TABLE table=rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.698838 16732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 479fcadbde0449f4bfeef0bd2f82ebaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.713312 16994 tablet_service.cc:1467] Processing CreateTablet for tablet bc9cc69a319d46b8940f4c7dc2e7e656 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.713778 16994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9cc69a319d46b8940f4c7dc2e7e656. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.715570 16339 tablet_service.cc:1467] Processing CreateTablet for tablet bc9cc69a319d46b8940f4c7dc2e7e656 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.715981 16339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9cc69a319d46b8940f4c7dc2e7e656. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.723426 16205 tablet_service.cc:1467] Processing CreateTablet for tablet bc9cc69a319d46b8940f4c7dc2e7e656 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.723876 16205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9cc69a319d46b8940f4c7dc2e7e656. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.724149 16469 tablet_service.cc:1467] Processing CreateTablet for tablet bc9cc69a319d46b8940f4c7dc2e7e656 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.724563 16469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9cc69a319d46b8940f4c7dc2e7e656. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.725064 16339 tablet_service.cc:1467] Processing CreateTablet for tablet dd96134671e1487eba32aaa1dcd6cd67 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.729496 16339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd96134671e1487eba32aaa1dcd6cd67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.729677 18433 tablet_bootstrap.cc:492] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:34.729759 16470 tablet_service.cc:1467] Processing CreateTablet for tablet dd96134671e1487eba32aaa1dcd6cd67 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.730134 16470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd96134671e1487eba32aaa1dcd6cd67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.731350 18433 tablet_bootstrap.cc:654] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.733275 16863 tablet_service.cc:1467] Processing CreateTablet for tablet dd96134671e1487eba32aaa1dcd6cd67 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.733505 16203 tablet_service.cc:1467] Processing CreateTablet for tablet dd96134671e1487eba32aaa1dcd6cd67 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.733680 16863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd96134671e1487eba32aaa1dcd6cd67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.733927 16203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd96134671e1487eba32aaa1dcd6cd67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.734252 18431 tablet_bootstrap.cc:492] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb: Bootstrap starting.
I20240502 01:55:34.735788 18431 tablet_bootstrap.cc:654] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.737584 18430 tablet_bootstrap.cc:492] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304: Bootstrap starting.
I20240502 01:55:34.738879 16733 tablet_service.cc:1467] Processing CreateTablet for tablet dd96134671e1487eba32aaa1dcd6cd67 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.739071 16602 tablet_service.cc:1467] Processing CreateTablet for tablet dd96134671e1487eba32aaa1dcd6cd67 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.739439 18436 tablet_bootstrap.cc:492] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:34.739457 16602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd96134671e1487eba32aaa1dcd6cd67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.739995 18435 tablet_bootstrap.cc:492] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9: Bootstrap starting.
I20240502 01:55:34.740314 16995 tablet_service.cc:1467] Processing CreateTablet for tablet dd96134671e1487eba32aaa1dcd6cd67 (DEFAULT_TABLE table=rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 01:55:34.740739 16995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd96134671e1487eba32aaa1dcd6cd67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.743034 16733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd96134671e1487eba32aaa1dcd6cd67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:34.745190 18436 tablet_bootstrap.cc:654] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.747087 18431 tablet_bootstrap.cc:492] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb: No bootstrap required, opened a new log
I20240502 01:55:34.748466 18430 tablet_bootstrap.cc:654] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.749020 18431 ts_tablet_manager.cc:1367] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240502 01:55:34.752642 18431 raft_consensus.cc:348] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.753676 18432 tablet_bootstrap.cc:492] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a: Bootstrap starting.
I20240502 01:55:34.755353 18432 tablet_bootstrap.cc:654] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.755632 18434 tablet_bootstrap.cc:492] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e: Bootstrap starting.
I20240502 01:55:34.758116 18433 tablet_bootstrap.cc:492] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367: No bootstrap required, opened a new log
I20240502 01:55:34.758339 18433 ts_tablet_manager.cc:1367] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20240502 01:55:34.759256 18434 tablet_bootstrap.cc:654] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.759673 18432 tablet_bootstrap.cc:492] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a: No bootstrap required, opened a new log
I20240502 01:55:34.759459 18433 raft_consensus.cc:348] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.759761 18433 raft_consensus.cc:374] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.759822 18432 ts_tablet_manager.cc:1367] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 01:55:34.759910 18433 raft_consensus.cc:729] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.760254 18433 consensus_queue.cc:260] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.760708 18435 tablet_bootstrap.cc:654] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.760548 18432 raft_consensus.cc:348] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.760797 18432 raft_consensus.cc:374] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.760890 18432 raft_consensus.cc:729] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.761186 18432 consensus_queue.cc:260] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.761914 18432 ts_tablet_manager.cc:1398] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 01:55:34.763008 18432 tablet_bootstrap.cc:492] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a: Bootstrap starting.
I20240502 01:55:34.763989 18434 tablet_bootstrap.cc:492] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e: No bootstrap required, opened a new log
I20240502 01:55:34.764185 18434 ts_tablet_manager.cc:1367] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 01:55:34.764541 18432 tablet_bootstrap.cc:654] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.764966 18434 raft_consensus.cc:348] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.765334 18434 raft_consensus.cc:374] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.765462 18434 raft_consensus.cc:729] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4d50250184223992325c86cb6c68e, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.765751 18434 consensus_queue.cc:260] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.771502 18430 tablet_bootstrap.cc:492] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304: No bootstrap required, opened a new log
I20240502 01:55:34.771757 18430 ts_tablet_manager.cc:1367] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20240502 01:55:34.772650 18430 raft_consensus.cc:348] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.773008 18430 raft_consensus.cc:374] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.773159 18430 raft_consensus.cc:729] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.773473 18430 consensus_queue.cc:260] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.774314 18430 ts_tablet_manager.cc:1398] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:34.774458 18433 ts_tablet_manager.cc:1398] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 01:55:34.776572 18434 ts_tablet_manager.cc:1398] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e: Time spent starting tablet: real 0.012s user 0.002s sys 0.003s
I20240502 01:55:34.777558 18433 tablet_bootstrap.cc:492] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:34.779251 18433 tablet_bootstrap.cc:654] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.780601 18431 raft_consensus.cc:374] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.780730 18431 raft_consensus.cc:729] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fcb8d12cb4ab28ec821c9473adeeb, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.780972 18431 consensus_queue.cc:260] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.782492 18434 tablet_bootstrap.cc:492] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e: Bootstrap starting.
I20240502 01:55:34.784237 18434 tablet_bootstrap.cc:654] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.786917 18432 tablet_bootstrap.cc:492] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a: No bootstrap required, opened a new log
I20240502 01:55:34.787168 18432 ts_tablet_manager.cc:1367] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.014s
I20240502 01:55:34.787973 18432 raft_consensus.cc:348] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:34.788342 18432 raft_consensus.cc:374] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.788444 18434 tablet_bootstrap.cc:492] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e: No bootstrap required, opened a new log
I20240502 01:55:34.788460 18432 raft_consensus.cc:729] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.788794 18434 ts_tablet_manager.cc:1367] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.001s
I20240502 01:55:34.788707 18432 consensus_queue.cc:260] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:34.789752 18435 tablet_bootstrap.cc:492] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9: No bootstrap required, opened a new log
I20240502 01:55:34.790114 18432 ts_tablet_manager.cc:1398] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:55:34.790143 18433 tablet_bootstrap.cc:492] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367: No bootstrap required, opened a new log
I20240502 01:55:34.790323 18433 ts_tablet_manager.cc:1367] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 01:55:34.790452 18432 tablet_bootstrap.cc:492] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a: Bootstrap starting.
I20240502 01:55:34.791486 18430 tablet_bootstrap.cc:492] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304: Bootstrap starting.
I20240502 01:55:34.792248 18432 tablet_bootstrap.cc:654] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.792263 18435 ts_tablet_manager.cc:1367] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.000s
I20240502 01:55:34.792397 18436 tablet_bootstrap.cc:492] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad: No bootstrap required, opened a new log
I20240502 01:55:34.792618 18436 ts_tablet_manager.cc:1367] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.003s
I20240502 01:55:34.793033 18430 tablet_bootstrap.cc:654] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.794628 18433 raft_consensus.cc:348] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:34.794915 18433 raft_consensus.cc:374] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.795001 18433 raft_consensus.cc:729] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.795264 18433 consensus_queue.cc:260] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:34.794833 18435 raft_consensus.cc:348] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:34.796160 18433 ts_tablet_manager.cc:1398] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:34.797475 18434 raft_consensus.cc:348] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:34.797794 18434 raft_consensus.cc:374] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.797859 18434 raft_consensus.cc:729] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4d50250184223992325c86cb6c68e, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.798134 18434 consensus_queue.cc:260] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:34.800524 18433 tablet_bootstrap.cc:492] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:34.802274 18434 ts_tablet_manager.cc:1398] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240502 01:55:34.805351 18435 raft_consensus.cc:374] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.805542 18435 raft_consensus.cc:729] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.793962 18436 raft_consensus.cc:348] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.806110 18434 tablet_bootstrap.cc:492] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e: Bootstrap starting.
I20240502 01:55:34.807314 18436 raft_consensus.cc:374] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.807498 18433 tablet_bootstrap.cc:654] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.807686 18434 tablet_bootstrap.cc:654] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.807839 18436 raft_consensus.cc:729] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.808106 18435 consensus_queue.cc:260] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:34.809231 18436 consensus_queue.cc:260] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.809697 18430 tablet_bootstrap.cc:492] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304: No bootstrap required, opened a new log
I20240502 01:55:34.809875 18430 ts_tablet_manager.cc:1367] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.014s
I20240502 01:55:34.810070 18435 ts_tablet_manager.cc:1398] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240502 01:55:34.810406 18435 tablet_bootstrap.cc:492] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9: Bootstrap starting.
I20240502 01:55:34.810583 18430 raft_consensus.cc:348] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.812454 18430 raft_consensus.cc:374] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.812577 18430 raft_consensus.cc:729] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.812955 18430 consensus_queue.cc:260] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.813706 18430 ts_tablet_manager.cc:1398] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:55:34.811452 18431 ts_tablet_manager.cc:1398] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent starting tablet: real 0.062s user 0.000s sys 0.003s
I20240502 01:55:34.814397 18431 tablet_bootstrap.cc:492] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb: Bootstrap starting.
I20240502 01:55:34.814636 18430 tablet_bootstrap.cc:492] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304: Bootstrap starting.
I20240502 01:55:34.816166 18430 tablet_bootstrap.cc:654] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.816586 18431 tablet_bootstrap.cc:654] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.811678 18434 tablet_bootstrap.cc:492] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e: No bootstrap required, opened a new log
I20240502 01:55:34.812023 18436 ts_tablet_manager.cc:1398] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20240502 01:55:34.817885 18434 ts_tablet_manager.cc:1367] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s
I20240502 01:55:34.818193 18436 tablet_bootstrap.cc:492] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:34.819000 18433 tablet_bootstrap.cc:492] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367: No bootstrap required, opened a new log
I20240502 01:55:34.818936 18434 raft_consensus.cc:348] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.819217 18434 raft_consensus.cc:374] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.819324 18434 raft_consensus.cc:729] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd4d50250184223992325c86cb6c68e, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.819566 18434 consensus_queue.cc:260] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.820164 18434 ts_tablet_manager.cc:1398] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:55:34.820407 18433 ts_tablet_manager.cc:1367] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240502 01:55:34.821285 18433 raft_consensus.cc:348] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.822881 18436 tablet_bootstrap.cc:654] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.824055 18435 tablet_bootstrap.cc:654] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.824193 18433 raft_consensus.cc:374] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.824340 18433 raft_consensus.cc:729] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.825258 18433 consensus_queue.cc:260] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.829794 18433 ts_tablet_manager.cc:1398] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 01:55:34.830181 18430 tablet_bootstrap.cc:492] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304: No bootstrap required, opened a new log
I20240502 01:55:34.830371 18430 ts_tablet_manager.cc:1367] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.009s
I20240502 01:55:34.831074 18430 raft_consensus.cc:348] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:34.831379 18430 raft_consensus.cc:374] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.831508 18430 raft_consensus.cc:729] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.831909 18435 tablet_bootstrap.cc:492] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9: No bootstrap required, opened a new log
I20240502 01:55:34.831909 18436 tablet_bootstrap.cc:492] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad: No bootstrap required, opened a new log
I20240502 01:55:34.831774 18430 consensus_queue.cc:260] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:34.832099 18436 ts_tablet_manager.cc:1367] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.002s
I20240502 01:55:34.832468 18435 ts_tablet_manager.cc:1367] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20240502 01:55:34.833138 18430 ts_tablet_manager.cc:1398] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:55:34.833238 18435 raft_consensus.cc:348] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.833294 18436 raft_consensus.cc:348] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:34.833534 18435 raft_consensus.cc:374] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.833563 18436 raft_consensus.cc:374] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.833627 18436 raft_consensus.cc:729] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.833632 18435 raft_consensus.cc:729] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.833876 18436 consensus_queue.cc:260] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:34.834625 18431 tablet_bootstrap.cc:492] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb: No bootstrap required, opened a new log
I20240502 01:55:34.834826 18431 ts_tablet_manager.cc:1367] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20240502 01:55:34.835597 18431 raft_consensus.cc:348] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:34.835911 18431 raft_consensus.cc:374] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.836004 18431 raft_consensus.cc:729] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fcb8d12cb4ab28ec821c9473adeeb, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.836252 18431 consensus_queue.cc:260] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:34.836887 18436 ts_tablet_manager.cc:1398] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 01:55:34.836884 18431 ts_tablet_manager.cc:1398] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:55:34.837208 18436 tablet_bootstrap.cc:492] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:34.837222 18431 tablet_bootstrap.cc:492] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb: Bootstrap starting.
I20240502 01:55:34.840822 18436 tablet_bootstrap.cc:654] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.841217 18435 consensus_queue.cc:260] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.841881 18431 tablet_bootstrap.cc:654] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.843142 18435 ts_tablet_manager.cc:1398] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 01:55:34.844710 18435 tablet_bootstrap.cc:492] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9: Bootstrap starting.
I20240502 01:55:34.845680 18432 tablet_bootstrap.cc:492] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a: No bootstrap required, opened a new log
I20240502 01:55:34.845861 18432 ts_tablet_manager.cc:1367] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a: Time spent bootstrapping tablet: real 0.055s user 0.021s sys 0.022s
I20240502 01:55:34.846010 18366 raft_consensus.cc:482] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:34.846174 18366 raft_consensus.cc:504] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.846864 18432 raft_consensus.cc:348] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.847199 18432 raft_consensus.cc:374] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.847318 18432 raft_consensus.cc:729] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.847455 18366 leader_election.cc:288] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:34.847568 18432 consensus_queue.cc:260] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.848151 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479fcadbde0449f4bfeef0bd2f82ebaf" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:34.848340 16359 raft_consensus.cc:2454] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:34.848376 18432 ts_tablet_manager.cc:1398] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:34.848448 17014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479fcadbde0449f4bfeef0bd2f82ebaf" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:34.848680 17014 raft_consensus.cc:2454] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:34.849118 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479fcadbde0449f4bfeef0bd2f82ebaf" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:34.849366 16885 raft_consensus.cc:2454] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:34.849512 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479fcadbde0449f4bfeef0bd2f82ebaf" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:34.849750 16619 raft_consensus.cc:2454] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:34.850817 16426 leader_election.cc:302] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 6b9f9aaa1c86420190af75270914449a, a6575ebf978a4631a13ba5c243c3c4f9, dc9fcb8d12cb4ab28ec821c9473adeeb; no voters:
I20240502 01:55:34.851155 18369 raft_consensus.cc:2781] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:34.851265 18369 raft_consensus.cc:482] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:34.851385 18369 raft_consensus.cc:3037] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.851493 18435 tablet_bootstrap.cc:654] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9: Neither blocks nor log segments found. Creating new log.
I20240502 01:55:34.851728 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479fcadbde0449f4bfeef0bd2f82ebaf" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:34.851927 16754 raft_consensus.cc:2454] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:34.852396 16223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479fcadbde0449f4bfeef0bd2f82ebaf" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:34.852627 16223 raft_consensus.cc:2454] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 0.
I20240502 01:55:34.853749 18436 tablet_bootstrap.cc:492] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad: No bootstrap required, opened a new log
I20240502 01:55:34.853935 18436 ts_tablet_manager.cc:1367] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.003s
I20240502 01:55:34.853997 18435 tablet_bootstrap.cc:492] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9: No bootstrap required, opened a new log
I20240502 01:55:34.854152 18435 ts_tablet_manager.cc:1367] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 01:55:34.853972 18369 raft_consensus.cc:504] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.854892 18435 raft_consensus.cc:348] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.854704 18436 raft_consensus.cc:348] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.855156 18436 raft_consensus.cc:374] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.855284 18436 raft_consensus.cc:729] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.855291 18369 leader_election.cc:288] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 election: Requested vote from peers cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:34.855620 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479fcadbde0449f4bfeef0bd2f82ebaf" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a"
I20240502 01:55:34.855552 18436 consensus_queue.cc:260] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.855836 16885 raft_consensus.cc:3037] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.856340 16223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479fcadbde0449f4bfeef0bd2f82ebaf" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e"
I20240502 01:55:34.856387 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479fcadbde0449f4bfeef0bd2f82ebaf" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad"
I20240502 01:55:34.856549 16223 raft_consensus.cc:3037] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.856565 16754 raft_consensus.cc:3037] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.856968 18436 ts_tablet_manager.cc:1398] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 01:55:34.858186 16885 raft_consensus.cc:2454] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:34.855712 17014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479fcadbde0449f4bfeef0bd2f82ebaf" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9"
I20240502 01:55:34.858466 17014 raft_consensus.cc:3037] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.859248 16754 raft_consensus.cc:2454] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:34.859995 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479fcadbde0449f4bfeef0bd2f82ebaf" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367"
I20240502 01:55:34.860234 16619 raft_consensus.cc:3037] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.860397 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "479fcadbde0449f4bfeef0bd2f82ebaf" candidate_uuid: "39059854b64045698ded1ba4e8d47304" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
I20240502 01:55:34.860589 16359 raft_consensus.cc:3037] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.860599 17014 raft_consensus.cc:2454] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:34.861332 18435 raft_consensus.cc:374] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.861433 18435 raft_consensus.cc:729] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6575ebf978a4631a13ba5c243c3c4f9, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.861646 16423 leader_election.cc:302] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 6b9f9aaa1c86420190af75270914449a, a6575ebf978a4631a13ba5c243c3c4f9, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:34.861660 18435 consensus_queue.cc:260] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.862489 18435 ts_tablet_manager.cc:1398] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 01:55:34.862833 16619 raft_consensus.cc:2454] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:34.862882 18152 raft_consensus.cc:482] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:34.863040 18152 raft_consensus.cc:504] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.864279 18152 leader_election.cc:288] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:34.864382 16357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd96134671e1487eba32aaa1dcd6cd67" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:34.864499 16359 raft_consensus.cc:2454] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
W20240502 01:55:34.865023 16690 leader_election.cc:341] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133): Illegal state: must be running to vote when last-logged opid is not known
I20240502 01:55:34.865391 17014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd96134671e1487eba32aaa1dcd6cd67" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:34.865625 17014 raft_consensus.cc:2454] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:34.865590 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd96134671e1487eba32aaa1dcd6cd67" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:34.865795 16490 raft_consensus.cc:2454] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:34.866154 16885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd96134671e1487eba32aaa1dcd6cd67" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:34.866370 16885 raft_consensus.cc:2454] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:34.866470 18366 raft_consensus.cc:2781] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:34.866636 18366 raft_consensus.cc:686] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Becoming Leader. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Running, Role: LEADER
I20240502 01:55:34.866779 16688 leader_election.cc:302] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 6b9f9aaa1c86420190af75270914449a, a6575ebf978a4631a13ba5c243c3c4f9, ed2e7e4188d0400b8689e28bbf5096ad; no voters: dc9fcb8d12cb4ab28ec821c9473adeeb
I20240502 01:55:34.866964 18366 consensus_queue.cc:237] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.867277 18152 raft_consensus.cc:2781] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:34.867386 18152 raft_consensus.cc:482] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:34.867478 18152 raft_consensus.cc:3037] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.866175 16224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd96134671e1487eba32aaa1dcd6cd67" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:34.868993 16224 raft_consensus.cc:2454] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:34.859263 16223 raft_consensus.cc:2454] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39059854b64045698ded1ba4e8d47304 in term 1.
I20240502 01:55:34.870721 18431 tablet_bootstrap.cc:492] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb: No bootstrap required, opened a new log
I20240502 01:55:34.870880 18431 ts_tablet_manager.cc:1367] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20240502 01:55:34.871670 18431 raft_consensus.cc:348] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.871219 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd96134671e1487eba32aaa1dcd6cd67" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367" is_pre_election: true
I20240502 01:55:34.871932 18431 raft_consensus.cc:374] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 01:55:34.872004 16619 raft_consensus.cc:2454] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 0.
I20240502 01:55:34.872031 18431 raft_consensus.cc:729] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc9fcb8d12cb4ab28ec821c9473adeeb, State: Initialized, Role: FOLLOWER
I20240502 01:55:34.872287 18431 consensus_queue.cc:260] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.872876 18431 ts_tablet_manager.cc:1398] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:34.872613 18152 raft_consensus.cc:504] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.874222 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd96134671e1487eba32aaa1dcd6cd67" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
I20240502 01:55:34.874292 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd96134671e1487eba32aaa1dcd6cd67" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304"
I20240502 01:55:34.874470 16490 raft_consensus.cc:3037] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.874513 16359 raft_consensus.cc:3037] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.874359 16079 catalog_manager.cc:5484] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39059854b64045698ded1ba4e8d47304 (127.31.161.195). New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:34.876024 16223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd96134671e1487eba32aaa1dcd6cd67" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e"
I20240502 01:55:34.876222 16223 raft_consensus.cc:3037] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.877398 16490 raft_consensus.cc:2454] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:34.878398 16883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd96134671e1487eba32aaa1dcd6cd67" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a"
I20240502 01:55:34.878610 16883 raft_consensus.cc:3037] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.878641 16619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd96134671e1487eba32aaa1dcd6cd67" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703f6f043c6041a4b629933690580367"
I20240502 01:55:34.878815 16619 raft_consensus.cc:3037] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.879684 16223 raft_consensus.cc:2454] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:34.881315 18152 leader_election.cc:288] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 election: Requested vote from peers dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:34.881390 16883 raft_consensus.cc:2454] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:34.881814 16619 raft_consensus.cc:2454] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:34.881870 17014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd96134671e1487eba32aaa1dcd6cd67" candidate_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9"
I20240502 01:55:34.882052 17014 raft_consensus.cc:3037] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.882267 16359 raft_consensus.cc:2454] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:34.882289 16690 leader_election.cc:302] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 703f6f043c6041a4b629933690580367, cdd4d50250184223992325c86cb6c68e, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:34.882562 18286 raft_consensus.cc:2781] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:34.882740 18286 raft_consensus.cc:686] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Becoming Leader. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Running, Role: LEADER
I20240502 01:55:34.882962 18286 consensus_queue.cc:237] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:34.884107 17014 raft_consensus.cc:2454] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed2e7e4188d0400b8689e28bbf5096ad in term 1.
I20240502 01:55:34.887171 16082 catalog_manager.cc:5484] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: term changed from 0 to 1, leader changed from <none> to ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197). New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:34.897850 18205 raft_consensus.cc:482] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 01:55:34.897989 18205 raft_consensus.cc:504] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:34.899133 18205 leader_election.cc:288] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:34.899396 16883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cc69a319d46b8940f4c7dc2e7e656" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a" is_pre_election: true
I20240502 01:55:34.899652 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cc69a319d46b8940f4c7dc2e7e656" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" is_pre_election: true
I20240502 01:55:34.899735 16223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cc69a319d46b8940f4c7dc2e7e656" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e" is_pre_election: true
I20240502 01:55:34.899791 17014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cc69a319d46b8940f4c7dc2e7e656" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" is_pre_election: true
I20240502 01:55:34.899886 16359 raft_consensus.cc:2454] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:34.899964 16223 raft_consensus.cc:2454] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:34.899997 17014 raft_consensus.cc:2454] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:34.899755 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cc69a319d46b8940f4c7dc2e7e656" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304" is_pre_election: true
I20240502 01:55:34.900281 16883 raft_consensus.cc:2454] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:34.900341 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cc69a319d46b8940f4c7dc2e7e656" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" is_pre_election: true
I20240502 01:55:34.900524 16490 raft_consensus.cc:2454] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:34.900528 16555 leader_election.cc:302] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 703f6f043c6041a4b629933690580367, a6575ebf978a4631a13ba5c243c3c4f9, cdd4d50250184223992325c86cb6c68e, dc9fcb8d12cb4ab28ec821c9473adeeb; no voters:
I20240502 01:55:34.900585 16754 raft_consensus.cc:2454] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 0.
I20240502 01:55:34.900887 18205 raft_consensus.cc:2781] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 01:55:34.901021 18205 raft_consensus.cc:482] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 01:55:34.901192 18205 raft_consensus.cc:3037] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.903311 18205 raft_consensus.cc:504] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:34.904348 18205 leader_election.cc:288] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 election: Requested vote from peers ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:34.904634 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cc69a319d46b8940f4c7dc2e7e656" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad"
I20240502 01:55:34.904742 16883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cc69a319d46b8940f4c7dc2e7e656" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9f9aaa1c86420190af75270914449a"
I20240502 01:55:34.904743 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cc69a319d46b8940f4c7dc2e7e656" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
I20240502 01:55:34.904783 17014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cc69a319d46b8940f4c7dc2e7e656" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9"
I20240502 01:55:34.904840 16754 raft_consensus.cc:3037] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.904920 16883 raft_consensus.cc:3037] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.904922 16359 raft_consensus.cc:3037] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.904963 17014 raft_consensus.cc:3037] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.907548 16883 raft_consensus.cc:2454] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
I20240502 01:55:34.907742 16754 raft_consensus.cc:2454] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
I20240502 01:55:34.908092 16490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cc69a319d46b8940f4c7dc2e7e656" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39059854b64045698ded1ba4e8d47304"
I20240502 01:55:34.908111 16223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc9cc69a319d46b8940f4c7dc2e7e656" candidate_uuid: "703f6f043c6041a4b629933690580367" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd4d50250184223992325c86cb6c68e"
I20240502 01:55:34.908285 16223 raft_consensus.cc:3037] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.908360 16490 raft_consensus.cc:3037] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 [term 0 FOLLOWER]: Advancing to term 1
I20240502 01:55:34.909828 16359 raft_consensus.cc:2454] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
I20240502 01:55:34.910152 16558 leader_election.cc:302] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 6b9f9aaa1c86420190af75270914449a, 703f6f043c6041a4b629933690580367, dc9fcb8d12cb4ab28ec821c9473adeeb, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:34.910444 18205 raft_consensus.cc:2781] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 01:55:34.910722 18205 raft_consensus.cc:686] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Becoming Leader. State: Replica: 703f6f043c6041a4b629933690580367, State: Running, Role: LEADER
I20240502 01:55:34.911028 16490 raft_consensus.cc:2454] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
I20240502 01:55:34.911041 16223 raft_consensus.cc:2454] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
I20240502 01:55:34.911095 18205 consensus_queue.cc:237] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:34.911607 17014 raft_consensus.cc:2454] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703f6f043c6041a4b629933690580367 in term 1.
I20240502 01:55:34.914880 16082 catalog_manager.cc:5484] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 reported cstate change: term changed from 0 to 1, leader changed from <none> to 703f6f043c6041a4b629933690580367 (127.31.161.196). New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:35.249459 18369 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:35.262601 18369 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:35.277608 18152 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:35.280750 18369 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:35.296339 18192 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:35.296793 18192 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:35.303531 18152 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:35.308494 18152 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:35.320075 18192 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:35.322948 18332 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:35.360020 18464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:35.360464 18464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 01:55:35.364826 18286 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:35.365329 18286 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 01:55:35.371251 18464 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 01:55:35.371837 18286 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
I20240502 01:55:35.429703 18332 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:35.459523 18332 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:35.470396 18332 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:35.470816 18332 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 01:55:35.486119 18332 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 436.586 ms
time per row: 0.218293 ms
I20240502 01:55:35.984310 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.201:44737
--local_ip_for_outbound_sockets=127.31.161.201
--tserver_master_addrs=127.31.161.254:46705
--webserver_port=46157
--webserver_interface=127.31.161.201
--builtin_ntp_servers=127.31.161.212:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 01:55:36.344784 18563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:36.345065 18563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:36.345237 18563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:36.353713 18563 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:36.353863 18563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:36.353992 18563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.201
I20240502 01:55:36.363806 18563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.201:44737
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.31.161.201
--webserver_port=46157
--tserver_master_addrs=127.31.161.254:46705
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.201
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:36.365602 18563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:36.367498 18563 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:36.382154 18570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:36.382565 18569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:36.382696 18572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:36.383378 18563 server_base.cc:1034] running on GCE node
I20240502 01:55:36.383997 18563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:36.385000 18563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:36.386220 18563 hybrid_clock.cc:648] HybridClock initialized: now 1714614936386162 us; error 47 us; skew 500 ppm
I20240502 01:55:36.386559 18563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:36.388818 18563 webserver.cc:460] Webserver started at http://127.31.161.201:46157/ using document root <none> and password file <none>
I20240502 01:55:36.389751 18563 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:36.389861 18563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:36.396091 18563 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 01:55:36.399101 18578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:36.400516 18563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 01:55:36.400682 18563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "a7c4573db86a415285d274ca57dde659"
format_stamp: "Formatted at 2024-05-02 01:55:18 on dist-test-slave-06l8"
I20240502 01:55:36.401659 18563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:36.421334 18563 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:55:36.422124 18563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:36.422391 18563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:36.423347 18563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:36.425146 18563 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:36.425238 18563 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20240502 01:55:36.425379 18563 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:36.425443 18563 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:36.472172 18563 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.201:44737
I20240502 01:55:36.472175 18691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.201:44737 every 8 connection(s)
I20240502 01:55:36.473627 18563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240502 01:55:36.478297 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 18563
I20240502 01:55:36.478902 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.202:43977
--local_ip_for_outbound_sockets=127.31.161.202
--tserver_master_addrs=127.31.161.254:46705
--webserver_port=41537
--webserver_interface=127.31.161.202
--builtin_ntp_servers=127.31.161.212:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:36.489656 18692 heartbeater.cc:347] Connected to a master server at 127.31.161.254:46705
I20240502 01:55:36.489974 18692 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:36.490891 18692 heartbeater.cc:511] Master 127.31.161.254:46705 requested a full tablet report, sending...
I20240502 01:55:36.492810 16082 ts_manager.cc:194] Re-registered known tserver with Master: a7c4573db86a415285d274ca57dde659 (127.31.161.201:44737)
I20240502 01:55:36.493759 16082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.201:37477
W20240502 01:55:36.857239 18696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:36.857602 18696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:36.857749 18696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:36.866022 18696 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:36.866147 18696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:36.866305 18696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.202
I20240502 01:55:36.875561 18696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.202:43977
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--webserver_interface=127.31.161.202
--webserver_port=41537
--tserver_master_addrs=127.31.161.254:46705
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.202
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:36.877177 18696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:36.879089 18696 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:36.891925 18703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:36.892446 18706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:36.892467 18704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:36.892910 18696 server_base.cc:1034] running on GCE node
I20240502 01:55:36.893464 18696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:36.894351 18696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:36.895550 18696 hybrid_clock.cc:648] HybridClock initialized: now 1714614936895489 us; error 40 us; skew 500 ppm
I20240502 01:55:36.895897 18696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:36.898550 18696 webserver.cc:460] Webserver started at http://127.31.161.202:41537/ using document root <none> and password file <none>
I20240502 01:55:36.899263 18696 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:36.899371 18696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:36.904386 18696 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20240502 01:55:36.907030 18712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:36.908317 18696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240502 01:55:36.908455 18696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/wal
uuid: "aa7ec9abef1747e5a9e2404cee265c20"
format_stamp: "Formatted at 2024-05-02 01:55:18 on dist-test-slave-06l8"
I20240502 01:55:36.909251 18696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:36.928323 18696 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:55:36.929121 18696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:36.929431 18696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:36.930294 18696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:36.931797 18696 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:36.931869 18696 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:36.931980 18696 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:36.932040 18696 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:36.981359 18696 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.202:43977
I20240502 01:55:36.981395 18825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.202:43977 every 8 connection(s)
I20240502 01:55:36.983004 18696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/data/info.pb
I20240502 01:55:36.984690 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 18696
I20240502 01:55:36.985447 32391 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMchxvY/build/asan/bin/kudu
/tmp/dist-test-taskMchxvY/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.161.200:38559
--local_ip_for_outbound_sockets=127.31.161.200
--tserver_master_addrs=127.31.161.254:46705
--webserver_port=35675
--webserver_interface=127.31.161.200
--builtin_ntp_servers=127.31.161.212:35895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 01:55:36.996992 18826 heartbeater.cc:347] Connected to a master server at 127.31.161.254:46705
I20240502 01:55:36.997290 18826 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:36.998061 18826 heartbeater.cc:511] Master 127.31.161.254:46705 requested a full tablet report, sending...
I20240502 01:55:36.999385 16082 ts_manager.cc:194] Re-registered known tserver with Master: aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202:43977)
I20240502 01:55:37.000209 16082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.202:48295
W20240502 01:55:37.344098 18829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 01:55:37.344388 18829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 01:55:37.344523 18829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 01:55:37.352080 18829 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 01:55:37.352195 18829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 01:55:37.352348 18829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.161.200
I20240502 01:55:37.361299 18829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.161.212:35895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.161.200:38559
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.31.161.200
--webserver_port=35675
--tserver_master_addrs=127.31.161.254:46705
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.161.200
--log_dir=/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 02 May 2024 01:43:22 UTC on efd1337a2aa3
build id 83
ASAN enabled
I20240502 01:55:37.362533 18829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 01:55:37.364223 18829 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 01:55:37.377184 18836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 01:55:37.377740 18839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:37.378553 18829 server_base.cc:1034] running on GCE node
W20240502 01:55:37.378549 18837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 01:55:37.379321 18829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 01:55:37.380120 18829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 01:55:37.381318 18829 hybrid_clock.cc:648] HybridClock initialized: now 1714614937381275 us; error 32 us; skew 500 ppm
I20240502 01:55:37.381634 18829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 01:55:37.383993 18829 webserver.cc:460] Webserver started at http://127.31.161.200:35675/ using document root <none> and password file <none>
I20240502 01:55:37.384771 18829 fs_manager.cc:359] Metadata directory not provided
I20240502 01:55:37.384922 18829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 01:55:37.390893 18829 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20240502 01:55:37.394140 18845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:37.395371 18829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240502 01:55:37.395519 18829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "0982e15b4c74425cad8d7d642eb63316"
format_stamp: "Formatted at 2024-05-02 01:55:17 on dist-test-slave-06l8"
I20240502 01:55:37.396322 18829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 01:55:37.422864 18829 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 01:55:37.423774 18829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 01:55:37.424063 18829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 01:55:37.425050 18829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 01:55:37.426596 18829 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 01:55:37.426678 18829 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:37.426796 18829 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 01:55:37.426848 18829 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 01:55:37.462064 18829 rpc_server.cc:307] RPC server started. Bound to: 127.31.161.200:38559
I20240502 01:55:37.462145 18958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.161.200:38559 every 8 connection(s)
I20240502 01:55:37.463559 18829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 01:55:37.473259 32391 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMchxvY/build/asan/bin/kudu as pid 18829
I20240502 01:55:37.476876 18959 heartbeater.cc:347] Connected to a master server at 127.31.161.254:46705
I20240502 01:55:37.477188 18959 heartbeater.cc:464] Registering TS with master...
I20240502 01:55:37.478195 18959 heartbeater.cc:511] Master 127.31.161.254:46705 requested a full tablet report, sending...
I20240502 01:55:37.479923 16082 ts_manager.cc:194] Re-registered known tserver with Master: 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200:38559)
I20240502 01:55:37.480829 16082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.161.200:34385
I20240502 01:55:37.496748 18692 heartbeater.cc:503] Master 127.31.161.254:46705 was elected leader, sending a full tablet report...
I20240502 01:55:38.002813 18826 heartbeater.cc:503] Master 127.31.161.254:46705 was elected leader, sending a full tablet report...
I20240502 01:55:38.007834 16471 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:55:38.047461 16602 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:38.061717 16995 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:38.065539 18893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:38.080631 16864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:38.081511 18760 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:38.082926 18626 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:38.097011 16204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:38.098474 16339 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:38.145481 16733 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:38.392354 18893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:38.400199 16471 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:55:38.414363 16864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:38.422371 16602 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:38.443127 18626 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:38.444869 18760 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:38.458787 16995 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 01:55:38.481666 16951 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:41953, user_credentials={real_user=slave}} blocked reactor thread for 58293.5us
I20240502 01:55:38.483292 18959 heartbeater.cc:503] Master 127.31.161.254:46705 was elected leader, sending a full tablet report...
I20240502 01:55:38.486490 16204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:38.512459 16339 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:38.557371 16733 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:38.834511 18893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:38.843442 16864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:38.849313 16602 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:38.855160 16471 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:55:38.863008 16995 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:38.908934 18626 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:38.923094 16204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:38.934609 16339 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:38.946874 18760 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:38.971686 16733 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:39.072242 16753 consensus_queue.cc:237] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:39.075424 16883 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.075538 17014 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.075647 16359 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.076306 18698 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.076637 18152 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.078872 19069 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.081909 16621 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.083557 19069 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.091259 16225 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.090902 19069 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.091948 18152 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.095240 16489 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.096009 19117 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.096303 17014 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.097153 16621 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.099181 16225 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.102949 16359 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.106637 16075 catalog_manager.cc:5484] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.106640 16883 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.118140 16489 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.135854 16753 consensus_queue.cc:237] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:39.139923 16225 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.140389 16489 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.141182 19124 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.141695 18698 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.141700 16883 consensus_queue.cc:237] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:39.145934 17014 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.146246 16621 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.146584 19124 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.146873 16359 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.147621 19124 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.148031 19124 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.149278 19117 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:39.152117 16621 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:39.153926 17014 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:39.158330 16359 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:39.159471 16225 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:39.159399 16489 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:39.160671 16067 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b55b98b06aca47988b14f486973cc370 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:39.166239 16077 catalog_manager.cc:5484] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:39.172719 16885 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.173360 18698 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.175022 16885 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
W20240502 01:55:39.183897 16688 consensus_peers.cc:485] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad -> Peer 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200:38559): Couldn't send request to peer 0982e15b4c74425cad8d7d642eb63316. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b55b98b06aca47988b14f486973cc370. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:39.184581 16754 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.185266 19109 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.185573 16621 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.185591 16359 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.185914 17014 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.186196 16489 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.186594 19011 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.187141 19011 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.187551 19139 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.189244 16225 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.189796 19109 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.198280 19139 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.199069 19109 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.199625 16754 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.205893 16075 catalog_manager.cc:5484] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "6b9f9aaa1c86420190af75270914449a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.208878 16621 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.215601 16223 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.221843 16359 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.223884 16883 consensus_queue.cc:237] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:39.228318 16359 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.228446 16621 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.229027 19109 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.229506 19109 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.234397 16223 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.234833 16754 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.234663 17014 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.235365 19146 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.235797 19146 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.237955 19146 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.238709 19109 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.242570 17016 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.242517 16223 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.248139 17016 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.248920 16068 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4da77826275d42b2b9e280c732575cfd with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:39.250213 16076 catalog_manager.cc:5484] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e reported cstate change: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "6b9f9aaa1c86420190af75270914449a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.254217 16621 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.254364 16359 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.257289 16490 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 01:55:39.258260 16820 consensus_peers.cc:485] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a -> Peer a7c4573db86a415285d274ca57dde659 (127.31.161.201:44737): Couldn't send request to peer a7c4573db86a415285d274ca57dde659. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4da77826275d42b2b9e280c732575cfd. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:39.258074 16753 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.258579 19011 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.259865 16490 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.264532 16490 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.285415 16621 consensus_queue.cc:237] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:39.288501 16223 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.288529 16490 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.289134 19019 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.289003 16359 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.289564 18963 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.290057 19019 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.291671 17016 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.292723 19019 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.302007 16883 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.302255 16753 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.302666 19019 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.302975 18963 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.310592 18963 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.310832 16359 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.311956 16489 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.315573 16225 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.317163 16075 catalog_manager.cc:5484] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.317577 17016 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.324571 16621 consensus_queue.cc:237] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:39.331377 17016 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.331992 19156 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.335139 16885 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.336339 16754 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.337194 19019 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.337859 19019 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.338382 16885 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.342067 16225 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.341660 16754 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.342741 19156 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.352891 19110 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
W20240502 01:55:39.357614 16557 consensus_peers.cc:485] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 -> Peer a7c4573db86a415285d274ca57dde659 (127.31.161.201:44737): Couldn't send request to peer a7c4573db86a415285d274ca57dde659. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6cde24f949334006aa9e241e844e81eb. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:39.362416 16359 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.363600 16489 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.359817 16885 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.364410 18963 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.365828 16359 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.367966 16753 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.365046 18963 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20240502 01:55:39.370501 16489 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.371444 16076 catalog_manager.cc:5484] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.381347 16070 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6cde24f949334006aa9e241e844e81eb with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:39.383348 17016 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.387477 16225 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.411926 16489 consensus_queue.cc:237] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:39.416116 16225 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.417447 16359 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.417682 16619 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.417675 18366 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.418071 19055 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.421126 18366 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20240502 01:55:39.429993 17016 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.430008 16753 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.430078 16885 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.430814 19055 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.431404 19055 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.431705 18366 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.437893 19055 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.441257 16619 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.448750 16076 catalog_manager.cc:5484] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.449600 16885 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.450783 16754 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.458786 16225 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.458808 16359 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.468677 16490 consensus_queue.cc:237] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:39.481591 16754 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.481586 16885 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.481585 16619 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.482206 19055 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.482699 19055 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.483400 18366 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.496169 16223 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.497717 16359 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.497242 19187 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.498504 19175 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.501461 19175 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.505605 16223 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.506659 16359 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
W20240502 01:55:39.514968 16425 consensus_peers.cc:485] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 -> Peer a7c4573db86a415285d274ca57dde659 (127.31.161.201:44737): Couldn't send request to peer a7c4573db86a415285d274ca57dde659. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48a81a0b7d47479e929d80041f34a3b1. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:39.520525 16076 catalog_manager.cc:5484] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e reported cstate change: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.521325 16067 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 48a81a0b7d47479e929d80041f34a3b1 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:39.525012 16754 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.525009 16885 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.535789 16621 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.547715 19189 ts_tablet_manager.cc:902] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316: Initiating tablet copy from peer ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825)
I20240502 01:55:39.558053 19189 tablet_copy_client.cc:308] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.197:46825
I20240502 01:55:39.568351 17014 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.569273 19188 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.570925 17014 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.575755 17014 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.578401 16774 tablet_copy_service.cc:140] P ed2e7e4188d0400b8689e28bbf5096ad: Received BeginTabletCopySession request for tablet b55b98b06aca47988b14f486973cc370 from peer 0982e15b4c74425cad8d7d642eb63316 ({username='slave'} at 127.31.161.200:49223)
I20240502 01:55:39.578648 16774 tablet_copy_service.cc:161] P ed2e7e4188d0400b8689e28bbf5096ad: Beginning new tablet copy session on tablet b55b98b06aca47988b14f486973cc370 from peer 0982e15b4c74425cad8d7d642eb63316 at {username='slave'} at 127.31.161.200:49223: session id = 0982e15b4c74425cad8d7d642eb63316-b55b98b06aca47988b14f486973cc370
I20240502 01:55:39.585285 16490 consensus_queue.cc:237] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:39.589479 16223 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.590229 16885 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.590909 19175 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.591951 16359 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.592746 16621 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.592809 19175 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.593624 19188 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.593855 19187 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.594339 17014 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.585913 16774 tablet_copy_source_session.cc:205] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:39.590528 16754 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.596026 19187 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.597476 19187 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.602303 19189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55b98b06aca47988b14f486973cc370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:39.607532 19175 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.615613 16754 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.616886 16223 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.619498 16359 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.619725 16885 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.621515 16621 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.622701 16075 catalog_manager.cc:5484] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:39.618992 17014 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.630795 19189 tablet_copy_client.cc:783] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:39.631197 19189 tablet_copy_client.cc:647] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:39.641820 19189 tablet_copy_client.cc:523] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:39.646224 16490 consensus_queue.cc:237] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:39.648626 19189 tablet_bootstrap.cc:492] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap starting.
I20240502 01:55:39.657826 16223 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.658515 19187 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:55:39.662635 16424 consensus_peers.cc:485] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 -> Peer 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200:38559): Couldn't send request to peer 0982e15b4c74425cad8d7d642eb63316. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 18e676bcd8b04f18a6814d6ebbd1b488. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:39.663349 16621 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.663928 19175 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.664330 16754 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.664999 19175 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.667098 16359 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.667917 19055 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.670614 17014 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.671430 19055 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:39.675981 16754 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:39.677211 19188 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.678853 16885 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.679613 17014 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:39.680323 16359 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:39.685344 16067 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 18e676bcd8b04f18a6814d6ebbd1b488 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:39.685240 16081 catalog_manager.cc:5484] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:39.684978 16223 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:39.685773 19187 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.688657 16885 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:39.693789 16621 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:39.699695 19189 log.cc:824] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:39.719539 16621 consensus_queue.cc:237] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:39.723173 16223 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.723277 16359 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.723842 16490 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.724009 16885 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.724498 17014 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.724730 19164 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.725220 16754 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.725306 19164 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.727190 19156 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.727666 19164 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.727908 18963 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.733230 19156 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.742501 19169 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.745522 16224 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.746129 16754 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.751791 16080 catalog_manager.cc:5484] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.751178 16490 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.760293 16885 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.761560 16359 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.763851 17014 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:39.771672 16621 consensus_queue.cc:237] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:39.783519 19189 tablet_bootstrap.cc:492] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:39.784092 19189 tablet_bootstrap.cc:492] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap complete.
I20240502 01:55:39.784621 19189 ts_tablet_manager.cc:1367] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316: Time spent bootstrapping tablet: real 0.136s user 0.058s sys 0.020s
W20240502 01:55:39.785869 16557 consensus_peers.cc:485] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 -> Peer a7c4573db86a415285d274ca57dde659 (127.31.161.201:44737): Couldn't send request to peer a7c4573db86a415285d274ca57dde659. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cec84c633b0a4c4fb6e074c79432fe61. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:39.785940 16754 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.786600 18963 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.789714 16885 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.790457 19164 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.790690 19189 raft_consensus.cc:348] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:39.791208 19189 raft_consensus.cc:729] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0982e15b4c74425cad8d7d642eb63316, State: Initialized, Role: LEARNER
I20240502 01:55:39.791826 19189 consensus_queue.cc:260] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:39.792650 16359 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.793263 19164 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.793886 16224 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.794514 19164 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.795894 16490 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.797020 19169 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.797459 17016 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.799401 16224 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.800570 16883 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.802320 19164 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.806478 18963 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20240502 01:55:39.810050 16080 catalog_manager.cc:5484] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a reported cstate change: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.810636 19189 ts_tablet_manager.cc:1398] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316: Time spent starting tablet: real 0.026s user 0.011s sys 0.012s
I20240502 01:55:39.811565 16774 tablet_copy_service.cc:342] P ed2e7e4188d0400b8689e28bbf5096ad: Request end of tablet copy session 0982e15b4c74425cad8d7d642eb63316-b55b98b06aca47988b14f486973cc370 received from {username='slave'} at 127.31.161.200:49223
I20240502 01:55:39.811759 16774 tablet_copy_service.cc:433] P ed2e7e4188d0400b8689e28bbf5096ad: ending tablet copy session 0982e15b4c74425cad8d7d642eb63316-b55b98b06aca47988b14f486973cc370 on tablet b55b98b06aca47988b14f486973cc370 with peer 0982e15b4c74425cad8d7d642eb63316
I20240502 01:55:39.814635 17014 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.822245 16490 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.823089 16070 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cec84c633b0a4c4fb6e074c79432fe61 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:39.823846 16754 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.824515 16359 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:39.837489 19224 ts_tablet_manager.cc:902] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659: Initiating tablet copy from peer 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121)
I20240502 01:55:39.841614 16754 consensus_queue.cc:237] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:39.842083 19224 tablet_copy_client.cc:308] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.198:38121
I20240502 01:55:39.845209 16621 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.845225 17014 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.845407 16885 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.846066 16490 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.846889 19069 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.847179 19124 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.847677 18698 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.848007 19124 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.848963 16224 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.850831 18698 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.852569 16905 tablet_copy_service.cc:140] P 6b9f9aaa1c86420190af75270914449a: Received BeginTabletCopySession request for tablet 4da77826275d42b2b9e280c732575cfd from peer a7c4573db86a415285d274ca57dde659 ({username='slave'} at 127.31.161.201:37183)
I20240502 01:55:39.852847 16905 tablet_copy_service.cc:161] P 6b9f9aaa1c86420190af75270914449a: Beginning new tablet copy session on tablet 4da77826275d42b2b9e280c732575cfd from peer a7c4573db86a415285d274ca57dde659 at {username='slave'} at 127.31.161.201:37183: session id = a7c4573db86a415285d274ca57dde659-4da77826275d42b2b9e280c732575cfd
I20240502 01:55:39.856966 16905 tablet_copy_source_session.cc:205] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:39.862062 16359 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:39.863534 18698 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.864749 19224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da77826275d42b2b9e280c732575cfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:39.887636 19069 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.889776 17014 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.897477 19224 tablet_copy_client.cc:783] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:39.897123 16077 catalog_manager.cc:5484] T 9eb6f3c696db4acf8a048f8ddc9e1230 P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.898097 19224 tablet_copy_client.cc:647] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:39.898684 16359 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.898736 16490 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.900108 19239 ts_tablet_manager.cc:902] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659: Initiating tablet copy from peer 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:39.904022 19224 tablet_copy_client.cc:523] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:39.906111 16223 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.909492 16754 consensus_queue.cc:237] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:39.910648 19224 tablet_bootstrap.cc:492] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659: Bootstrap starting.
I20240502 01:55:39.920874 16224 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.921461 18698 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.922135 16359 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.925031 18698 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.926084 19224 log.cc:824] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:39.927101 16490 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.927677 18698 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.931675 18698 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:39.936542 16489 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:39.942020 16067 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9eb6f3c696db4acf8a048f8ddc9e1230 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:39.944304 16077 catalog_manager.cc:5484] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:39.949245 16621 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.952646 19069 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.954077 16621 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.961169 17014 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.961737 19069 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:39.963213 17014 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:39.968470 19247 ts_tablet_manager.cc:902] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659: Initiating tablet copy from peer 703f6f043c6041a4b629933690580367 (127.31.161.196:40425)
I20240502 01:55:39.972434 16223 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:39.972398 16359 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:39.979768 19239 tablet_copy_client.cc:308] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:35693
I20240502 01:55:39.989305 16883 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:39.992513 19243 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:39.994503 16511 tablet_copy_service.cc:140] P 39059854b64045698ded1ba4e8d47304: Received BeginTabletCopySession request for tablet 48a81a0b7d47479e929d80041f34a3b1 from peer a7c4573db86a415285d274ca57dde659 ({username='slave'} at 127.31.161.201:59931)
I20240502 01:55:39.994112 16883 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:39.994748 16511 tablet_copy_service.cc:161] P 39059854b64045698ded1ba4e8d47304: Beginning new tablet copy session on tablet 48a81a0b7d47479e929d80041f34a3b1 from peer a7c4573db86a415285d274ca57dde659 at {username='slave'} at 127.31.161.201:59931: session id = a7c4573db86a415285d274ca57dde659-48a81a0b7d47479e929d80041f34a3b1
I20240502 01:55:39.998911 16883 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:40.003675 16511 tablet_copy_source_session.cc:205] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:40.004882 16621 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:40.007514 19239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a81a0b7d47479e929d80041f34a3b1. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:55:40.014659 16688 consensus_peers.cc:485] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad -> Peer aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202:43977): Couldn't send request to peer aa7ec9abef1747e5a9e2404cee265c20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9eb6f3c696db4acf8a048f8ddc9e1230. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:40.015578 19247 tablet_copy_client.cc:308] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:40425
I20240502 01:55:40.021220 19239 tablet_copy_client.cc:783] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:40.021637 19239 tablet_copy_client.cc:647] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:40.022296 16642 tablet_copy_service.cc:140] P 703f6f043c6041a4b629933690580367: Received BeginTabletCopySession request for tablet 6cde24f949334006aa9e241e844e81eb from peer a7c4573db86a415285d274ca57dde659 ({username='slave'} at 127.31.161.201:36131)
I20240502 01:55:40.022567 16642 tablet_copy_service.cc:161] P 703f6f043c6041a4b629933690580367: Beginning new tablet copy session on tablet 6cde24f949334006aa9e241e844e81eb from peer a7c4573db86a415285d274ca57dde659 at {username='slave'} at 127.31.161.201:36131: session id = a7c4573db86a415285d274ca57dde659-6cde24f949334006aa9e241e844e81eb
I20240502 01:55:40.023975 19239 tablet_copy_client.cc:523] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:40.025929 16642 tablet_copy_source_session.cc:205] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:40.028817 19239 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659: Bootstrap starting.
I20240502 01:55:40.029325 17014 consensus_queue.cc:237] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:40.033658 16359 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:40.034596 19247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cde24f949334006aa9e241e844e81eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:40.034569 16754 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:40.035106 19067 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:40.037406 19065 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:40.040400 16223 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:40.040920 16490 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:40.041363 19065 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:40.041627 19067 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:40.046286 19247 tablet_copy_client.cc:783] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:40.046770 19247 tablet_copy_client.cc:647] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:40.049280 19247 tablet_copy_client.cc:523] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:40.052742 19247 tablet_bootstrap.cc:492] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659: Bootstrap starting.
I20240502 01:55:40.053153 19189 ts_tablet_manager.cc:902] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316: Initiating tablet copy from peer 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:40.054523 16621 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:40.055466 19065 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:40.058045 16883 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:40.061252 19065 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:40.063416 19065 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:40.067829 16359 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:40.070396 16754 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:40.074257 16080 catalog_manager.cc:5484] T 53e266d2e3b446c3a50ed3b93a6a5dde P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:40.082198 19224 tablet_bootstrap.cc:492] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:40.082760 19224 tablet_bootstrap.cc:492] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659: Bootstrap complete.
I20240502 01:55:40.083256 19224 ts_tablet_manager.cc:1367] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659: Time spent bootstrapping tablet: real 0.173s user 0.075s sys 0.020s
I20240502 01:55:40.093425 19224 raft_consensus.cc:348] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:40.094303 19224 raft_consensus.cc:729] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a7c4573db86a415285d274ca57dde659, State: Initialized, Role: LEARNER
I20240502 01:55:40.094906 19224 consensus_queue.cc:260] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:40.095223 16223 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:40.098733 17014 consensus_queue.cc:237] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:40.108976 16754 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:40.109841 19067 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:40.112787 16223 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:40.114094 19067 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:40.122046 16489 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:40.124270 16359 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:40.125926 19067 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:40.127318 16489 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:40.133147 17014 consensus_queue.cc:237] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:40.137013 16753 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:40.137964 16620 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:40.138316 16884 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:40.138535 19067 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:40.139532 16223 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:40.140193 19283 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:40.140787 16359 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:40.142762 19278 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:40.143608 19278 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:40.146685 19278 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:40.147799 18913 raft_consensus.cc:1203] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:55:40.155709 16885 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:40.188416 19065 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20240502 01:55:40.192437 16357 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:40.193172 19282 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:40.194717 16882 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:40.201040 16080 catalog_manager.cc:5484] T 53e266d2e3b446c3a50ed3b93a6a5dde P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:40.208065 16882 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:40.191771 19283 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:40.221871 16067 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 53e266d2e3b446c3a50ed3b93a6a5dde with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:40.223227 16224 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:40.230895 19239 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:40.237452 19239 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659: Bootstrap complete.
I20240502 01:55:40.238523 19239 ts_tablet_manager.cc:1367] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659: Time spent bootstrapping tablet: real 0.210s user 0.057s sys 0.000s
W20240502 01:55:40.239023 16949 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.197:46825, user_credentials={real_user=slave}} blocked reactor thread for 101487us
I20240502 01:55:40.240242 16489 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:40.241837 16753 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:40.243248 19239 raft_consensus.cc:348] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:40.244243 19239 raft_consensus.cc:729] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a7c4573db86a415285d274ca57dde659, State: Initialized, Role: LEARNER
I20240502 01:55:40.244654 19239 consensus_queue.cc:260] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:40.251068 19067 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:40.255254 16489 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:40.255736 19067 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:40.262912 16359 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:40.265393 16884 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:40.268653 16753 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:40.270352 16080 catalog_manager.cc:5484] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:40.273483 19239 ts_tablet_manager.cc:1398] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659: Time spent starting tablet: real 0.034s user 0.004s sys 0.000s
I20240502 01:55:40.248220 19247 tablet_bootstrap.cc:492] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:40.274422 19247 tablet_bootstrap.cc:492] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659: Bootstrap complete.
I20240502 01:55:40.275360 19247 ts_tablet_manager.cc:1367] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659: Time spent bootstrapping tablet: real 0.223s user 0.039s sys 0.017s
I20240502 01:55:40.276073 16511 tablet_copy_service.cc:342] P 39059854b64045698ded1ba4e8d47304: Request end of tablet copy session a7c4573db86a415285d274ca57dde659-48a81a0b7d47479e929d80041f34a3b1 received from {username='slave'} at 127.31.161.201:59931
I20240502 01:55:40.276263 16511 tablet_copy_service.cc:433] P 39059854b64045698ded1ba4e8d47304: ending tablet copy session a7c4573db86a415285d274ca57dde659-48a81a0b7d47479e929d80041f34a3b1 on tablet 48a81a0b7d47479e929d80041f34a3b1 with peer a7c4573db86a415285d274ca57dde659
I20240502 01:55:40.277159 19247 raft_consensus.cc:348] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:40.277539 19247 raft_consensus.cc:729] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a7c4573db86a415285d274ca57dde659, State: Initialized, Role: LEARNER
I20240502 01:55:40.279110 19247 consensus_queue.cc:260] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:40.281200 19247 ts_tablet_manager.cc:1398] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 01:55:40.281989 16642 tablet_copy_service.cc:342] P 703f6f043c6041a4b629933690580367: Request end of tablet copy session a7c4573db86a415285d274ca57dde659-6cde24f949334006aa9e241e844e81eb received from {username='slave'} at 127.31.161.201:36131
I20240502 01:55:40.282186 16642 tablet_copy_service.cc:433] P 703f6f043c6041a4b629933690580367: ending tablet copy session a7c4573db86a415285d274ca57dde659-6cde24f949334006aa9e241e844e81eb on tablet 6cde24f949334006aa9e241e844e81eb with peer a7c4573db86a415285d274ca57dde659
I20240502 01:55:40.307340 16490 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:40.316689 18647 raft_consensus.cc:1203] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:55:40.326272 16223 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:40.336373 19247 ts_tablet_manager.cc:902] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659: Initiating tablet copy from peer 703f6f043c6041a4b629933690580367 (127.31.161.196:40425)
I20240502 01:55:40.340745 19224 ts_tablet_manager.cc:1398] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659: Time spent starting tablet: real 0.257s user 0.042s sys 0.076s
I20240502 01:55:40.342820 19189 tablet_copy_client.cc:308] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:35693
I20240502 01:55:40.342839 17016 consensus_queue.cc:237] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:40.345598 16619 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:40.350459 16223 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:40.350932 16905 tablet_copy_service.cc:342] P 6b9f9aaa1c86420190af75270914449a: Request end of tablet copy session a7c4573db86a415285d274ca57dde659-4da77826275d42b2b9e280c732575cfd received from {username='slave'} at 127.31.161.201:37183
I20240502 01:55:40.351119 16905 tablet_copy_service.cc:433] P 6b9f9aaa1c86420190af75270914449a: ending tablet copy session a7c4573db86a415285d274ca57dde659-4da77826275d42b2b9e280c732575cfd on tablet 4da77826275d42b2b9e280c732575cfd with peer a7c4573db86a415285d274ca57dde659
I20240502 01:55:40.351066 19283 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:40.355041 16359 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:40.355690 19067 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:40.360014 19282 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:40.365532 16619 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:40.372515 16884 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:40.374823 19282 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:55:40.376384 16950 consensus_peers.cc:485] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 -> Peer aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202:43977): Couldn't send request to peer aa7ec9abef1747e5a9e2404cee265c20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53e266d2e3b446c3a50ed3b93a6a5dde. This is attempt 1: this message will repeat every 5th retry.
W20240502 01:55:40.376585 16950 consensus_peers.cc:485] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 -> Peer aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202:43977): Couldn't send request to peer aa7ec9abef1747e5a9e2404cee265c20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 297561b66dd7472ab817c2a23723565c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:40.377801 16621 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:40.378358 19282 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:40.381438 16619 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:40.387800 16621 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:40.397539 16490 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:40.398106 16753 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:40.399730 19067 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:40.407183 16067 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 297561b66dd7472ab817c2a23723565c with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:40.408046 16620 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:40.408804 16224 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:40.409346 16884 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:40.414288 18647 raft_consensus.cc:1203] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:55:40.415473 16079 catalog_manager.cc:5484] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e reported cstate change: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:40.417449 16359 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:40.433645 19065 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:40.434204 19065 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:40.435745 16490 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:40.438094 16753 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:40.453119 16511 tablet_copy_service.cc:140] P 39059854b64045698ded1ba4e8d47304: Received BeginTabletCopySession request for tablet 18e676bcd8b04f18a6814d6ebbd1b488 from peer 0982e15b4c74425cad8d7d642eb63316 ({username='slave'} at 127.31.161.200:41079)
I20240502 01:55:40.453321 16511 tablet_copy_service.cc:161] P 39059854b64045698ded1ba4e8d47304: Beginning new tablet copy session on tablet 18e676bcd8b04f18a6814d6ebbd1b488 from peer 0982e15b4c74425cad8d7d642eb63316 at {username='slave'} at 127.31.161.200:41079: session id = 0982e15b4c74425cad8d7d642eb63316-18e676bcd8b04f18a6814d6ebbd1b488
I20240502 01:55:40.453735 19247 tablet_copy_client.cc:308] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:40425
I20240502 01:55:40.454649 16642 tablet_copy_service.cc:140] P 703f6f043c6041a4b629933690580367: Received BeginTabletCopySession request for tablet cec84c633b0a4c4fb6e074c79432fe61 from peer a7c4573db86a415285d274ca57dde659 ({username='slave'} at 127.31.161.201:36131)
I20240502 01:55:40.454831 16642 tablet_copy_service.cc:161] P 703f6f043c6041a4b629933690580367: Beginning new tablet copy session on tablet cec84c633b0a4c4fb6e074c79432fe61 from peer a7c4573db86a415285d274ca57dde659 at {username='slave'} at 127.31.161.201:36131: session id = a7c4573db86a415285d274ca57dde659-cec84c633b0a4c4fb6e074c79432fe61
I20240502 01:55:40.456435 16511 tablet_copy_source_session.cc:205] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:40.457737 16642 tablet_copy_source_session.cc:205] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:40.458089 19189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18e676bcd8b04f18a6814d6ebbd1b488. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:40.459861 19247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec84c633b0a4c4fb6e074c79432fe61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:40.464496 19247 tablet_copy_client.cc:783] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:40.464702 19247 tablet_copy_client.cc:647] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:40.469377 19189 tablet_copy_client.cc:783] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:40.469623 19189 tablet_copy_client.cc:647] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:40.471192 19189 tablet_copy_client.cc:523] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:40.472046 19247 tablet_copy_client.cc:523] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:40.481976 19247 tablet_bootstrap.cc:492] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659: Bootstrap starting.
I20240502 01:55:40.483206 19189 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap starting.
I20240502 01:55:40.519587 19310 ts_tablet_manager.cc:902] T 9eb6f3c696db4acf8a048f8ddc9e1230 P aa7ec9abef1747e5a9e2404cee265c20: Initiating tablet copy from peer ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825)
I20240502 01:55:40.555171 18698 raft_consensus.cc:1050] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad: attempting to promote NON_VOTER 0982e15b4c74425cad8d7d642eb63316 to VOTER
I20240502 01:55:40.556643 18698 consensus_queue.cc:237] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:40.562958 19247 tablet_bootstrap.cc:492] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:40.563462 19247 tablet_bootstrap.cc:492] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659: Bootstrap complete.
I20240502 01:55:40.563854 19247 ts_tablet_manager.cc:1367] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659: Time spent bootstrapping tablet: real 0.082s user 0.054s sys 0.016s
I20240502 01:55:40.564739 19247 raft_consensus.cc:348] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:40.565090 19247 raft_consensus.cc:729] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a7c4573db86a415285d274ca57dde659, State: Initialized, Role: LEARNER
I20240502 01:55:40.565330 19247 consensus_queue.cc:260] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:40.566002 19247 ts_tablet_manager.cc:1398] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 01:55:40.566715 16642 tablet_copy_service.cc:342] P 703f6f043c6041a4b629933690580367: Request end of tablet copy session a7c4573db86a415285d274ca57dde659-cec84c633b0a4c4fb6e074c79432fe61 received from {username='slave'} at 127.31.161.201:36131
I20240502 01:55:40.566839 16642 tablet_copy_service.cc:433] P 703f6f043c6041a4b629933690580367: ending tablet copy session a7c4573db86a415285d274ca57dde659-cec84c633b0a4c4fb6e074c79432fe61 on tablet cec84c633b0a4c4fb6e074c79432fe61 with peer a7c4573db86a415285d274ca57dde659
I20240502 01:55:40.581705 19189 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:40.582185 19189 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap complete.
I20240502 01:55:40.582496 19189 ts_tablet_manager.cc:1367] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316: Time spent bootstrapping tablet: real 0.099s user 0.050s sys 0.012s
I20240502 01:55:40.583575 19189 raft_consensus.cc:348] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:40.583926 19189 raft_consensus.cc:729] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0982e15b4c74425cad8d7d642eb63316, State: Initialized, Role: LEARNER
I20240502 01:55:40.584167 19189 consensus_queue.cc:260] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:40.589565 19189 ts_tablet_manager.cc:1398] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240502 01:55:40.590749 19317 ts_tablet_manager.cc:902] T 53e266d2e3b446c3a50ed3b93a6a5dde P aa7ec9abef1747e5a9e2404cee265c20: Initiating tablet copy from peer a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:40.590837 16511 tablet_copy_service.cc:342] P 39059854b64045698ded1ba4e8d47304: Request end of tablet copy session 0982e15b4c74425cad8d7d642eb63316-18e676bcd8b04f18a6814d6ebbd1b488 received from {username='slave'} at 127.31.161.200:41079
I20240502 01:55:40.590989 16511 tablet_copy_service.cc:433] P 39059854b64045698ded1ba4e8d47304: ending tablet copy session 0982e15b4c74425cad8d7d642eb63316-18e676bcd8b04f18a6814d6ebbd1b488 on tablet 18e676bcd8b04f18a6814d6ebbd1b488 with peer 0982e15b4c74425cad8d7d642eb63316
I20240502 01:55:40.595542 16621 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:40.595777 18913 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:40.596122 17016 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:40.596171 19069 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:40.596452 19243 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:40.597126 19243 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:40.606770 19310 tablet_copy_client.cc:308] T 9eb6f3c696db4acf8a048f8ddc9e1230 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.197:46825
I20240502 01:55:40.659577 18912 raft_consensus.cc:1203] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:55:40.662930 16882 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:40.663049 16224 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:40.663491 19243 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:40.663638 16774 tablet_copy_service.cc:140] P ed2e7e4188d0400b8689e28bbf5096ad: Received BeginTabletCopySession request for tablet 9eb6f3c696db4acf8a048f8ddc9e1230 from peer aa7ec9abef1747e5a9e2404cee265c20 ({username='slave'} at 127.31.161.202:42599)
I20240502 01:55:40.663825 16774 tablet_copy_service.cc:161] P ed2e7e4188d0400b8689e28bbf5096ad: Beginning new tablet copy session on tablet 9eb6f3c696db4acf8a048f8ddc9e1230 from peer aa7ec9abef1747e5a9e2404cee265c20 at {username='slave'} at 127.31.161.202:42599: session id = aa7ec9abef1747e5a9e2404cee265c20-9eb6f3c696db4acf8a048f8ddc9e1230
I20240502 01:55:40.667512 16359 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:40.667035 16774 tablet_copy_source_session.cc:205] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:40.671926 19243 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:40.677672 19324 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:40.685849 19310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb6f3c696db4acf8a048f8ddc9e1230. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:55:40.732151 16687 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.199:46837, user_credentials={real_user=slave}} blocked reactor thread for 72134.7us
I20240502 01:55:40.733052 16490 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:40.737674 19069 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:40.748220 18647 raft_consensus.cc:1203] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:55:40.758261 16223 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:40.765540 16082 catalog_manager.cc:5484] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e reported cstate change: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:40.757540 19336 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:40.804917 16490 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:40.805490 18913 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:40.806906 17016 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:40.809626 16621 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:40.822567 19310 tablet_copy_client.cc:783] T 9eb6f3c696db4acf8a048f8ddc9e1230 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:40.823652 16882 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:40.824198 16359 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:40.830106 19310 tablet_copy_client.cc:647] T 9eb6f3c696db4acf8a048f8ddc9e1230 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:40.831435 19317 tablet_copy_client.cc:308] T 53e266d2e3b446c3a50ed3b93a6a5dde P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.199:46837
I20240502 01:55:40.833551 19310 tablet_copy_client.cc:523] T 9eb6f3c696db4acf8a048f8ddc9e1230 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:40.835235 16753 consensus_queue.cc:237] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:40.838392 16223 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:40.838524 16359 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:40.838939 19069 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:40.839507 16619 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:40.839486 19069 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:40.840283 19335 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:40.841332 19310 tablet_bootstrap.cc:492] T 9eb6f3c696db4acf8a048f8ddc9e1230 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap starting.
I20240502 01:55:40.844120 17016 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:40.844825 19335 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:40.847621 18913 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:40.848433 19069 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:40.849500 19324 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:40.849498 17036 tablet_copy_service.cc:140] P a6575ebf978a4631a13ba5c243c3c4f9: Received BeginTabletCopySession request for tablet 53e266d2e3b446c3a50ed3b93a6a5dde from peer aa7ec9abef1747e5a9e2404cee265c20 ({username='slave'} at 127.31.161.202:59627)
I20240502 01:55:40.849711 17036 tablet_copy_service.cc:161] P a6575ebf978a4631a13ba5c243c3c4f9: Beginning new tablet copy session on tablet 53e266d2e3b446c3a50ed3b93a6a5dde from peer aa7ec9abef1747e5a9e2404cee265c20 at {username='slave'} at 127.31.161.202:59627: session id = aa7ec9abef1747e5a9e2404cee265c20-53e266d2e3b446c3a50ed3b93a6a5dde
I20240502 01:55:40.849575 16619 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:40.851125 17016 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:40.855193 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b55b98b06aca47988b14f486973cc370 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:40.855746 16359 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:40.855996 16223 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:40.855906 18913 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:40.857345 17036 tablet_copy_source_session.cc:205] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:40.859323 19317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53e266d2e3b446c3a50ed3b93a6a5dde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:40.861835 16080 catalog_manager.cc:5484] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e reported cstate change: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:40.867369 16864 tablet_service.cc:1514] Processing DeleteTablet for tablet b55b98b06aca47988b14f486973cc370 with delete_type TABLET_DATA_TOMBSTONED (TS 6b9f9aaa1c86420190af75270914449a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49752
I20240502 01:55:40.868145 19347 ts_tablet_manager.cc:902] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20: Initiating tablet copy from peer a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:40.870975 19169 raft_consensus.cc:1050] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367: attempting to promote NON_VOTER a7c4573db86a415285d274ca57dde659 to VOTER
I20240502 01:55:40.872293 19169 consensus_queue.cc:237] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:40.879956 19348 tablet_replica.cc:330] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a: stopping tablet replica
I20240502 01:55:40.880455 19348 raft_consensus.cc:2229] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:40.881033 19348 raft_consensus.cc:2258] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:40.884827 19317 tablet_copy_client.cc:783] T 53e266d2e3b446c3a50ed3b93a6a5dde P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:40.885097 19317 tablet_copy_client.cc:647] T 53e266d2e3b446c3a50ed3b93a6a5dde P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:40.887173 16490 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:40.887339 19317 tablet_copy_client.cc:523] T 53e266d2e3b446c3a50ed3b93a6a5dde P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:40.887814 19336 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:40.889604 16489 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:40.890903 19317 tablet_bootstrap.cc:492] T 53e266d2e3b446c3a50ed3b93a6a5dde P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap starting.
I20240502 01:55:40.891955 19310 log.cc:824] T 9eb6f3c696db4acf8a048f8ddc9e1230 P aa7ec9abef1747e5a9e2404cee265c20: Log is configured to *not* fsync() on all Append() calls
I20240502 01:55:40.893723 19348 ts_tablet_manager.cc:1875] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:40.900907 19348 ts_tablet_manager.cc:1888] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:40.901155 19348 log.cc:1196] T b55b98b06aca47988b14f486973cc370 P 6b9f9aaa1c86420190af75270914449a: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal/wals/b55b98b06aca47988b14f486973cc370
I20240502 01:55:40.902429 16068 catalog_manager.cc:4830] TS 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121): tablet b55b98b06aca47988b14f486973cc370 (table rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]) successfully deleted
I20240502 01:55:40.903697 19188 raft_consensus.cc:1050] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304: attempting to promote NON_VOTER a7c4573db86a415285d274ca57dde659 to VOTER
I20240502 01:55:40.905709 19188 consensus_queue.cc:237] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:40.922513 18647 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:40.923391 19175 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:40.933007 16359 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:40.933951 19175 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:40.957049 16224 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:40.958297 19175 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:40.971118 16753 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:40.971800 19187 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:40.975497 17016 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:40.978066 19187 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:40.986827 16621 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:40.988986 19187 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.011579 16882 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.012379 19187 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.018313 19317 tablet_bootstrap.cc:492] T 53e266d2e3b446c3a50ed3b93a6a5dde P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:41.019125 19317 tablet_bootstrap.cc:492] T 53e266d2e3b446c3a50ed3b93a6a5dde P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap complete.
I20240502 01:55:41.020412 19317 ts_tablet_manager.cc:1367] T 53e266d2e3b446c3a50ed3b93a6a5dde P aa7ec9abef1747e5a9e2404cee265c20: Time spent bootstrapping tablet: real 0.130s user 0.048s sys 0.014s
I20240502 01:55:41.027849 19317 raft_consensus.cc:348] T 53e266d2e3b446c3a50ed3b93a6a5dde P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:41.036064 19188 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.037401 19317 raft_consensus.cc:729] T 53e266d2e3b446c3a50ed3b93a6a5dde P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa7ec9abef1747e5a9e2404cee265c20, State: Initialized, Role: LEARNER
I20240502 01:55:41.038406 19317 consensus_queue.cc:260] T 53e266d2e3b446c3a50ed3b93a6a5dde P aa7ec9abef1747e5a9e2404cee265c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:41.042295 16359 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.044948 19317 ts_tablet_manager.cc:1398] T 53e266d2e3b446c3a50ed3b93a6a5dde P aa7ec9abef1747e5a9e2404cee265c20: Time spent starting tablet: real 0.024s user 0.002s sys 0.008s
I20240502 01:55:41.047415 17036 tablet_copy_service.cc:342] P a6575ebf978a4631a13ba5c243c3c4f9: Request end of tablet copy session aa7ec9abef1747e5a9e2404cee265c20-53e266d2e3b446c3a50ed3b93a6a5dde received from {username='slave'} at 127.31.161.202:59627
I20240502 01:55:41.047580 17036 tablet_copy_service.cc:433] P a6575ebf978a4631a13ba5c243c3c4f9: ending tablet copy session aa7ec9abef1747e5a9e2404cee265c20-53e266d2e3b446c3a50ed3b93a6a5dde on tablet 53e266d2e3b446c3a50ed3b93a6a5dde with peer aa7ec9abef1747e5a9e2404cee265c20
I20240502 01:55:41.054523 16077 catalog_manager.cc:5484] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.073612 18646 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.075381 18646 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.091706 16754 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.098623 19368 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.101430 16884 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.116192 19175 raft_consensus.cc:1050] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304: attempting to promote NON_VOTER 0982e15b4c74425cad8d7d642eb63316 to VOTER
I20240502 01:55:41.117478 19175 consensus_queue.cc:237] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:41.128926 16224 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.130702 17014 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.131282 19164 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.133514 19364 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.135991 18646 raft_consensus.cc:1203] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:55:41.140868 16490 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.143510 19368 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.144022 16359 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.145548 19368 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.164023 16224 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.173763 17014 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.179242 16621 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.183614 16359 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.184141 18780 raft_consensus.cc:1203] T 53e266d2e3b446c3a50ed3b93a6a5dde P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:55:41.184199 19187 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.184604 18913 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.186977 16490 consensus_queue.cc:237] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:41.192793 19292 raft_consensus.cc:1050] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a: attempting to promote NON_VOTER a7c4573db86a415285d274ca57dde659 to VOTER
I20240502 01:55:41.194113 19292 consensus_queue.cc:237] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:41.197818 19187 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.199163 18645 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.205523 16359 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.208652 19187 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.210831 19187 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.211438 16882 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.217996 18645 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.218797 17015 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.219350 19150 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.219961 19150 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.222482 19355 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.223397 17015 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.224469 16884 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.224843 16754 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.226806 16754 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.227406 19355 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.228385 19381 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.228860 16224 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.229452 19381 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.230279 19381 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.231055 19381 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.234416 16619 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.235162 19150 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.236758 16489 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.257966 19383 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.263206 16225 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.263895 19150 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.264343 16357 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.266536 19150 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.273720 16619 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.275400 19187 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240502 01:55:41.281877 16753 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.282464 19150 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.288617 19355 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.292896 16223 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.300034 16077 catalog_manager.cc:5484] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e reported cstate change: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.303128 16222 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.303735 19381 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.307951 18647 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.311581 16620 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.312197 19381 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.315832 17014 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.316033 16749 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.316483 19381 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.316924 19381 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.318725 19385 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.321033 17016 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.323114 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 48a81a0b7d47479e929d80041f34a3b1 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:41.324407 18646 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.326825 16082 catalog_manager.cc:5484] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.332146 16864 tablet_service.cc:1514] Processing DeleteTablet for tablet 48a81a0b7d47479e929d80041f34a3b1 with delete_type TABLET_DATA_TOMBSTONED (TS 6b9f9aaa1c86420190af75270914449a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49752
I20240502 01:55:41.332711 19348 tablet_replica.cc:330] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: stopping tablet replica
I20240502 01:55:41.331802 16357 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.332998 19348 raft_consensus.cc:2229] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:41.333520 19348 raft_consensus.cc:2258] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:41.336946 19348 ts_tablet_manager.cc:1875] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:41.342207 16749 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.343217 19348 ts_tablet_manager.cc:1888] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:41.343377 19348 log.cc:1196] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal/wals/48a81a0b7d47479e929d80041f34a3b1
I20240502 01:55:41.344190 16068 catalog_manager.cc:4830] TS 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121): tablet 48a81a0b7d47479e929d80041f34a3b1 (table rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]) successfully deleted
I20240502 01:55:41.351842 19381 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:41.352905 17014 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.353740 16620 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.354122 19292 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.356704 19310 tablet_bootstrap.cc:492] T 9eb6f3c696db4acf8a048f8ddc9e1230 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:41.359715 16752 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:41.362900 16751 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.363942 18913 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:41.370380 19310 tablet_bootstrap.cc:492] T 9eb6f3c696db4acf8a048f8ddc9e1230 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap complete.
I20240502 01:55:41.372435 17013 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.372570 18647 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.374027 16077 catalog_manager.cc:5484] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316 reported cstate change: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:41.370703 19310 ts_tablet_manager.cc:1367] T 9eb6f3c696db4acf8a048f8ddc9e1230 P aa7ec9abef1747e5a9e2404cee265c20: Time spent bootstrapping tablet: real 0.530s user 0.122s sys 0.131s
I20240502 01:55:41.378744 19310 raft_consensus.cc:348] T 9eb6f3c696db4acf8a048f8ddc9e1230 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:41.379041 19310 raft_consensus.cc:729] T 9eb6f3c696db4acf8a048f8ddc9e1230 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa7ec9abef1747e5a9e2404cee265c20, State: Initialized, Role: LEARNER
I20240502 01:55:41.379284 19310 consensus_queue.cc:260] T 9eb6f3c696db4acf8a048f8ddc9e1230 P aa7ec9abef1747e5a9e2404cee265c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:41.379882 19310 ts_tablet_manager.cc:1398] T 9eb6f3c696db4acf8a048f8ddc9e1230 P aa7ec9abef1747e5a9e2404cee265c20: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:55:41.379832 16359 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.380954 16225 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.381186 16754 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.386987 16080 catalog_manager.cc:5484] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e reported cstate change: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6b9f9aaa1c86420190af75270914449a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.387455 16882 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.390919 16774 tablet_copy_service.cc:342] P ed2e7e4188d0400b8689e28bbf5096ad: Request end of tablet copy session aa7ec9abef1747e5a9e2404cee265c20-9eb6f3c696db4acf8a048f8ddc9e1230 received from {username='slave'} at 127.31.161.202:42599
I20240502 01:55:41.391062 16774 tablet_copy_service.cc:433] P ed2e7e4188d0400b8689e28bbf5096ad: ending tablet copy session aa7ec9abef1747e5a9e2404cee265c20-9eb6f3c696db4acf8a048f8ddc9e1230 on tablet 9eb6f3c696db4acf8a048f8ddc9e1230 with peer aa7ec9abef1747e5a9e2404cee265c20
I20240502 01:55:41.377964 16489 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.377067 16621 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.398231 16620 consensus_queue.cc:237] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:41.404235 16359 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.404812 19367 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.406863 18645 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.407838 19367 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.408911 16225 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:41.410148 16224 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.410804 19367 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.411291 16223 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.415936 19164 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.417404 16224 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.419409 16883 consensus_queue.cc:237] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:41.423926 16070 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6cde24f949334006aa9e241e844e81eb with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:41.426079 16753 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.426330 16359 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.426654 19150 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.426939 18644 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.430058 19383 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.434022 16080 catalog_manager.cc:5484] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659 reported cstate change: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.438805 16733 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cde24f949334006aa9e241e844e81eb with delete_type TABLET_DATA_TOMBSTONED (TS ed2e7e4188d0400b8689e28bbf5096ad not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47194
W20240502 01:55:41.454099 16951 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:41953, user_credentials={real_user=slave}} blocked reactor thread for 92064.8us
I20240502 01:55:41.456141 17013 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.460443 16488 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.461385 18645 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.462201 19382 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.464906 19382 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.466898 19150 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.468789 17016 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.469489 19355 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.471307 16068 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4da77826275d42b2b9e280c732575cfd with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:41.471531 17016 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
W20240502 01:55:41.474061 16424 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.200:38559, user_credentials={real_user=slave}} blocked reactor thread for 100612us
I20240502 01:55:41.474344 17015 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.477703 16488 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.479032 16358 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.483151 16885 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:41.486426 17013 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:41.487084 16488 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.487655 19383 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.489917 16488 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.490819 16359 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:41.494259 19408 tablet_replica.cc:330] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad: stopping tablet replica
I20240502 01:55:41.494834 19408 raft_consensus.cc:2229] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:41.495468 19408 raft_consensus.cc:2258] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:41.499498 19375 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.501904 16359 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.507608 16077 catalog_manager.cc:5484] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "6b9f9aaa1c86420190af75270914449a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.513118 16602 tablet_service.cc:1514] Processing DeleteTablet for tablet 4da77826275d42b2b9e280c732575cfd with delete_type TABLET_DATA_TOMBSTONED (TS 703f6f043c6041a4b629933690580367 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53048
I20240502 01:55:41.514055 16884 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.514889 16488 consensus_queue.cc:237] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:41.516058 19355 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.517692 16884 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.519034 16224 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.519707 19382 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.520947 16751 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.521442 16224 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.522678 16357 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.527874 19414 tablet_replica.cc:330] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367: stopping tablet replica
I20240502 01:55:41.528561 19414 raft_consensus.cc:2229] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:41.528992 19414 raft_consensus.cc:2258] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:41.529999 16620 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:41.533510 19347 tablet_copy_client.cc:308] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.199:46837
I20240502 01:55:41.534237 17036 tablet_copy_service.cc:140] P a6575ebf978a4631a13ba5c243c3c4f9: Received BeginTabletCopySession request for tablet 297561b66dd7472ab817c2a23723565c from peer aa7ec9abef1747e5a9e2404cee265c20 ({username='slave'} at 127.31.161.202:59627)
I20240502 01:55:41.534426 17036 tablet_copy_service.cc:161] P a6575ebf978a4631a13ba5c243c3c4f9: Beginning new tablet copy session on tablet 297561b66dd7472ab817c2a23723565c from peer aa7ec9abef1747e5a9e2404cee265c20 at {username='slave'} at 127.31.161.202:59627: session id = aa7ec9abef1747e5a9e2404cee265c20-297561b66dd7472ab817c2a23723565c
I20240502 01:55:41.534479 18913 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.534855 16223 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.535337 19187 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.537333 17036 tablet_copy_source_session.cc:205] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:41.540133 19414 ts_tablet_manager.cc:1875] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:41.541924 17015 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.543440 16358 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.544317 19187 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.545863 19187 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.545979 19347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 297561b66dd7472ab817c2a23723565c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:41.547178 19187 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.549633 16883 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.551635 19414 ts_tablet_manager.cc:1888] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:41.551541 16488 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.555541 19414 log.cc:1196] T 4da77826275d42b2b9e280c732575cfd P 703f6f043c6041a4b629933690580367: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/4da77826275d42b2b9e280c732575cfd
I20240502 01:55:41.555821 19408 ts_tablet_manager.cc:1875] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:41.556473 16070 catalog_manager.cc:4830] TS 703f6f043c6041a4b629933690580367 (127.31.161.196:40425): tablet 4da77826275d42b2b9e280c732575cfd (table rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]) successfully deleted
I20240502 01:55:41.557739 19347 tablet_copy_client.cc:783] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:41.558004 19347 tablet_copy_client.cc:647] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:41.560757 19347 tablet_copy_client.cc:523] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:41.562611 18645 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.563685 19403 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:41.565167 19347 tablet_bootstrap.cc:492] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap starting.
I20240502 01:55:41.568886 16224 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:41.572046 17016 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:41.573171 16359 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:41.575457 16082 catalog_manager.cc:5484] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e reported cstate change: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:41.580086 16602 tablet_service.cc:1514] Processing DeleteTablet for tablet 18e676bcd8b04f18a6814d6ebbd1b488 with delete_type TABLET_DATA_TOMBSTONED (TS 703f6f043c6041a4b629933690580367 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53048
I20240502 01:55:41.580476 19414 tablet_replica.cc:330] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367: stopping tablet replica
I20240502 01:55:41.580691 19414 raft_consensus.cc:2229] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:41.580973 19414 raft_consensus.cc:2258] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:41.582098 19414 ts_tablet_manager.cc:1875] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:41.582356 19403 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.583137 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 18e676bcd8b04f18a6814d6ebbd1b488 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:41.584326 18913 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:41.588366 19414 ts_tablet_manager.cc:1888] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:41.588536 19414 log.cc:1196] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/18e676bcd8b04f18a6814d6ebbd1b488
I20240502 01:55:41.589383 16070 catalog_manager.cc:4830] TS 703f6f043c6041a4b629933690580367 (127.31.161.196:40425): tablet 18e676bcd8b04f18a6814d6ebbd1b488 (table rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]) successfully deleted
I20240502 01:55:41.591423 19408 ts_tablet_manager.cc:1888] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:41.591720 19408 log.cc:1196] T 6cde24f949334006aa9e241e844e81eb P ed2e7e4188d0400b8689e28bbf5096ad: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal/wals/6cde24f949334006aa9e241e844e81eb
I20240502 01:55:41.592913 16067 catalog_manager.cc:4830] TS ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825): tablet 6cde24f949334006aa9e241e844e81eb (table rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]) successfully deleted
I20240502 01:55:41.600956 16884 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:41.605939 16751 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.606596 19403 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.610000 16754 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:41.617038 19410 raft_consensus.cc:1050] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9: attempting to promote NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 to VOTER
I20240502 01:55:41.618188 19410 consensus_queue.cc:237] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:41.622797 16488 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.622787 16359 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.623378 19065 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.623986 19282 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.624318 18780 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.626174 19282 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.628896 16224 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.629528 19282 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.630771 16621 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.636420 19284 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.649483 16884 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.650148 19065 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.679752 16754 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.680478 19410 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.682837 19375 raft_consensus.cc:1050] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367: attempting to promote NON_VOTER a7c4573db86a415285d274ca57dde659 to VOTER
I20240502 01:55:41.685875 18780 raft_consensus.cc:1203] T 9eb6f3c696db4acf8a048f8ddc9e1230 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:55:41.684034 19375 consensus_queue.cc:237] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:41.693792 19065 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:41.695145 16882 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:41.696997 19347 tablet_bootstrap.cc:492] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:41.696607 16223 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:41.697461 19347 tablet_bootstrap.cc:492] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap complete.
I20240502 01:55:41.697767 19347 ts_tablet_manager.cc:1367] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20: Time spent bootstrapping tablet: real 0.133s user 0.065s sys 0.022s
I20240502 01:55:41.698659 19347 raft_consensus.cc:348] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:41.698989 19347 raft_consensus.cc:729] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa7ec9abef1747e5a9e2404cee265c20, State: Initialized, Role: LEARNER
I20240502 01:55:41.699213 19347 consensus_queue.cc:260] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:41.699934 19347 ts_tablet_manager.cc:1398] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:41.699479 16359 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:41.701875 17036 tablet_copy_service.cc:342] P a6575ebf978a4631a13ba5c243c3c4f9: Request end of tablet copy session aa7ec9abef1747e5a9e2404cee265c20-297561b66dd7472ab817c2a23723565c received from {username='slave'} at 127.31.161.202:59627
I20240502 01:55:41.701995 17036 tablet_copy_service.cc:433] P a6575ebf978a4631a13ba5c243c3c4f9: ending tablet copy session aa7ec9abef1747e5a9e2404cee265c20-297561b66dd7472ab817c2a23723565c on tablet 297561b66dd7472ab817c2a23723565c with peer aa7ec9abef1747e5a9e2404cee265c20
I20240502 01:55:41.703539 18780 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:41.704697 16082 catalog_manager.cc:5484] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:41.716444 16488 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:41.720968 16619 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:41.728840 16754 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:41.732034 17016 consensus_queue.cc:237] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:41.733412 18647 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.733995 19169 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.737027 16490 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.737288 16359 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.737634 19065 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.737887 19284 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.738333 18780 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.738885 19284 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.743363 16224 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.743319 19065 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:41.745595 16884 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.746302 19282 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.747656 16490 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:41.748003 16223 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:41.749621 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 53e266d2e3b446c3a50ed3b93a6a5dde with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:41.752297 16488 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.753629 16359 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.753587 16077 catalog_manager.cc:5484] T 53e266d2e3b446c3a50ed3b93a6a5dde P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:41.754166 19358 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.754827 17015 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.755107 19284 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.755378 16884 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.756003 19367 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.756474 19358 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.757362 16882 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:41.759212 19367 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.761314 16602 tablet_service.cc:1514] Processing DeleteTablet for tablet 53e266d2e3b446c3a50ed3b93a6a5dde with delete_type TABLET_DATA_TOMBSTONED (TS 703f6f043c6041a4b629933690580367 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53048
I20240502 01:55:41.760962 16359 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:41.761695 19414 tablet_replica.cc:330] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367: stopping tablet replica
I20240502 01:55:41.761902 19414 raft_consensus.cc:2229] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:41.762164 19414 raft_consensus.cc:2258] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:41.763850 18780 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:41.765281 19414 ts_tablet_manager.cc:1875] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:41.769290 16752 raft_consensus.cc:1261] T 53e266d2e3b446c3a50ed3b93a6a5dde P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.771239 19410 consensus_queue.cc:1035] T 53e266d2e3b446c3a50ed3b93a6a5dde P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.771683 16751 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.772284 19358 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.775915 16751 raft_consensus.cc:2932] T 53e266d2e3b446c3a50ed3b93a6a5dde P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:41.779258 19414 ts_tablet_manager.cc:1888] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:41.779642 19414 log.cc:1196] T 53e266d2e3b446c3a50ed3b93a6a5dde P 703f6f043c6041a4b629933690580367: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/53e266d2e3b446c3a50ed3b93a6a5dde
I20240502 01:55:41.780519 16070 catalog_manager.cc:4830] TS 703f6f043c6041a4b629933690580367 (127.31.161.196:40425): tablet 53e266d2e3b446c3a50ed3b93a6a5dde (table rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]) successfully deleted
I20240502 01:55:41.785946 16223 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:41.786684 19358 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:41.791520 19358 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.794245 16883 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.794342 16752 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.794409 16488 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.800068 16077 catalog_manager.cc:5484] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.802470 18780 raft_consensus.cc:1203] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:55:41.806128 18647 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.807996 16357 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.808975 17015 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.816841 16223 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.818593 16619 consensus_queue.cc:237] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:41.822157 16488 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.823159 19358 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.823560 18647 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.824769 16883 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.826500 17016 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.827908 16359 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.828168 19169 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.828411 16224 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:41.828998 19358 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.829327 19355 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.830133 19375 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.832893 19375 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:41.835026 19375 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.835732 16883 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.837663 18647 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.839897 16070 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cec84c633b0a4c4fb6e074c79432fe61 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:41.841642 16359 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.841789 16488 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.842495 16082 catalog_manager.cc:5484] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a reported cstate change: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.842394 16223 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.852980 16733 tablet_service.cc:1514] Processing DeleteTablet for tablet cec84c633b0a4c4fb6e074c79432fe61 with delete_type TABLET_DATA_TOMBSTONED (TS ed2e7e4188d0400b8689e28bbf5096ad not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47194
I20240502 01:55:41.853438 19408 tablet_replica.cc:330] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad: stopping tablet replica
I20240502 01:55:41.853658 19408 raft_consensus.cc:2229] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:41.854138 19408 raft_consensus.cc:2258] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:41.854084 17016 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:41.855175 19408 ts_tablet_manager.cc:1875] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:41.862107 19408 ts_tablet_manager.cc:1888] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:41.862313 19408 log.cc:1196] T cec84c633b0a4c4fb6e074c79432fe61 P ed2e7e4188d0400b8689e28bbf5096ad: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal/wals/cec84c633b0a4c4fb6e074c79432fe61
I20240502 01:55:41.863152 16067 catalog_manager.cc:4830] TS ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825): tablet cec84c633b0a4c4fb6e074c79432fe61 (table rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]) successfully deleted
I20240502 01:55:42.146939 16752 consensus_queue.cc:237] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:42.150367 16619 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.150674 16488 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.151345 17016 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.151288 19335 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.152097 16883 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.152688 19336 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.153178 19336 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.157876 18698 raft_consensus.cc:1050] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad: attempting to promote NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 to VOTER
I20240502 01:55:42.158998 18698 consensus_queue.cc:237] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:42.163934 19336 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.189694 16223 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.190156 16359 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.190601 19335 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.191149 19335 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.198969 19336 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:42.201269 17015 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:42.201786 16884 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:42.208492 16082 catalog_manager.cc:5484] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:42.213824 18780 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:42.215145 19335 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:42.221805 16359 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:42.231415 16751 consensus_queue.cc:237] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:42.236454 16488 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
W20240502 01:55:42.240123 16688 consensus_peers.cc:485] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad -> Peer 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200:38559): Couldn't send request to peer 0982e15b4c74425cad8d7d642eb63316. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd96134671e1487eba32aaa1dcd6cd67. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:42.240340 16883 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.240542 16359 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.242697 19335 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:42.243602 19335 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:42.257751 16490 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.262764 19474 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:42.265516 19335 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:42.268743 16490 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:42.275048 16080 catalog_manager.cc:5484] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:42.281468 16621 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.283082 17016 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:42.288811 16224 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.298722 19468 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.299192 19473 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.298835 19477 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:42.309173 16883 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:42.316015 16883 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:42.321116 16619 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:42.322356 16223 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:42.328877 16359 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:42.329670 19473 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 01:55:42.334312 16687 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.199:46837, user_credentials={real_user=slave}} blocked reactor thread for 50778.5us
I20240502 01:55:42.336704 16490 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:42.335909 19473 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:42.336383 16223 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:42.337986 16067 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet dd96134671e1487eba32aaa1dcd6cd67 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:42.341800 17016 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.343245 16359 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:42.345649 16619 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:42.351590 19479 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:42.352175 19477 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:42.354873 17015 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:42.352175 19479 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20240502 01:55:42.369530 19468 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:42.375465 19429 raft_consensus.cc:1050] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9: attempting to promote NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 to VOTER
I20240502 01:55:42.376669 19429 consensus_queue.cc:237] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:42.380089 18780 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:42.380539 16620 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:42.380833 16884 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:42.381121 19284 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:42.381871 19284 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:42.391428 16488 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:42.391819 16357 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:42.391912 16225 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:42.393544 19284 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:42.394119 19284 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:42.394565 19284 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:42.395538 16619 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:42.400974 19429 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:42.403995 16752 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:42.407730 19429 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:42.406157 16223 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:42.415618 16620 consensus_queue.cc:237] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:42.422487 16357 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.421934 19065 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.423046 19355 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.421598 19468 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.428643 16358 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.429775 16621 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.432623 16489 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.433398 19169 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.432768 16619 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.433804 16884 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.434403 19169 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.437450 16883 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.438971 18779 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.438706 16884 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.443395 16356 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.445782 16080 catalog_manager.cc:5484] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:42.447799 16077 catalog_manager.cc:5484] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 703f6f043c6041a4b629933690580367 reported cstate change: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.449432 18780 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.454653 16223 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.458801 19355 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.472751 17016 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.473448 19355 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.481586 16224 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.494789 16490 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.509766 16752 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.510566 19355 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.510356 16223 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.513790 19355 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:42.524279 16075 catalog_manager.cc:5484] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:42.526648 16224 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:42.528508 16359 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:42.522675 16488 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:42.530823 16620 consensus_queue.cc:237] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:42.541990 17013 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.539994 16883 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:42.553005 16751 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.554405 19169 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.554296 16489 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.555982 16751 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:42.556963 16222 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.557880 19169 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:42.574375 17016 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.575894 19505 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.576882 17014 consensus_queue.cc:237] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:42.581327 16882 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:42.578523 17015 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:42.582095 19065 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:42.588450 16359 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.588912 18780 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:42.589052 19516 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:42.589612 19065 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:42.598729 19169 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:42.599514 17016 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
W20240502 01:55:42.602849 16952 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:38133, user_credentials={real_user=slave}} blocked reactor thread for 149452us
I20240502 01:55:42.603686 16070 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f53b466fba6a4655a63a559b0ea56f77 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:42.606160 16491 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.606990 19516 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:42.606799 16359 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:42.609622 16620 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:42.612855 16488 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:42.613577 16082 catalog_manager.cc:5484] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:42.617543 16223 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:42.619346 19508 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:42.624050 16882 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.625380 16486 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:42.625278 16752 consensus_queue.cc:237] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:42.626322 16754 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:42.629966 19169 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:42.631203 19065 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:42.631649 19065 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.636082 16883 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:42.637745 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 297561b66dd7472ab817c2a23723565c with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:42.639547 16224 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:42.640422 19429 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:42.643159 16224 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.645658 17014 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:42.646219 19473 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:42.646665 16359 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:42.646946 16620 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.648119 16488 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:42.649052 16359 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:42.649127 19473 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:42.649266 16222 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:42.649586 19473 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:42.652141 19480 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:42.653108 19065 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:42.653726 19480 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.654809 16359 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.656404 16357 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.660744 16882 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.662595 16490 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.662395 16080 catalog_manager.cc:5484] T 9eb6f3c696db4acf8a048f8ddc9e1230 P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.657816 18779 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.667181 16222 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.673305 18780 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:42.673357 16082 catalog_manager.cc:5484] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 reported cstate change: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.673877 19474 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:42.678325 16733 tablet_service.cc:1514] Processing DeleteTablet for tablet 297561b66dd7472ab817c2a23723565c with delete_type TABLET_DATA_TOMBSTONED (TS ed2e7e4188d0400b8689e28bbf5096ad not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47194
I20240502 01:55:42.679168 16620 raft_consensus.cc:1261] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:42.681614 19474 consensus_queue.cc:1035] T 9eb6f3c696db4acf8a048f8ddc9e1230 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:42.683526 16620 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.684870 19525 tablet_replica.cc:330] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad: stopping tablet replica
I20240502 01:55:42.685144 19525 raft_consensus.cc:2229] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:42.685395 19525 pending_rounds.cc:62] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad: Trying to abort 1 pending ops.
I20240502 01:55:42.685796 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9eb6f3c696db4acf8a048f8ddc9e1230 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:42.687202 17016 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.687592 16488 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.689718 18780 raft_consensus.cc:2932] T 9eb6f3c696db4acf8a048f8ddc9e1230 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:42.694546 19525 pending_rounds.cc:68] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7023062803971366912 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "297561b66dd7472ab817c2a23723565c" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } } }
W20240502 01:55:42.695250 16556 consensus_peers.cc:485] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 -> Peer aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202:43977): Couldn't send request to peer aa7ec9abef1747e5a9e2404cee265c20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f53b466fba6a4655a63a559b0ea56f77. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:42.696805 19525 raft_consensus.cc:2894] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240502 01:55:42.697012 19525 raft_consensus.cc:2866] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 01:55:42.697239 19525 raft_consensus.cc:2258] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:42.698590 19525 ts_tablet_manager.cc:1875] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:42.713089 19525 ts_tablet_manager.cc:1888] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:42.713335 19525 log.cc:1196] T 297561b66dd7472ab817c2a23723565c P ed2e7e4188d0400b8689e28bbf5096ad: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal/wals/297561b66dd7472ab817c2a23723565c
I20240502 01:55:42.714030 16864 tablet_service.cc:1514] Processing DeleteTablet for tablet 9eb6f3c696db4acf8a048f8ddc9e1230 with delete_type TABLET_DATA_TOMBSTONED (TS 6b9f9aaa1c86420190af75270914449a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49752
I20240502 01:55:42.722005 16067 catalog_manager.cc:4830] TS ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825): tablet 297561b66dd7472ab817c2a23723565c (table rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]) successfully deleted
I20240502 01:55:42.722076 19526 tablet_replica.cc:330] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a: stopping tablet replica
I20240502 01:55:42.722466 19526 raft_consensus.cc:2229] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:42.722780 19526 raft_consensus.cc:2258] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:42.725751 19526 ts_tablet_manager.cc:1875] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:42.732530 19526 ts_tablet_manager.cc:1888] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:42.732692 19526 log.cc:1196] T 9eb6f3c696db4acf8a048f8ddc9e1230 P 6b9f9aaa1c86420190af75270914449a: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal/wals/9eb6f3c696db4acf8a048f8ddc9e1230
I20240502 01:55:42.733650 16068 catalog_manager.cc:4830] TS 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121): tablet 9eb6f3c696db4acf8a048f8ddc9e1230 (table rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]) successfully deleted
I20240502 01:55:42.741243 17016 consensus_queue.cc:237] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:42.746491 16883 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.746974 16357 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.747167 16752 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.747691 19522 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.748003 19284 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.748167 16490 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.748329 16223 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.748730 19284 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.749027 19520 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.757653 16620 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.760181 19522 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.766192 19522 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:42.767686 16883 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:42.767891 16490 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:42.771494 16223 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:42.771863 19522 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.771606 16752 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:42.775322 16075 catalog_manager.cc:5484] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:42.784336 17016 consensus_queue.cc:237] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:42.789923 16223 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.790047 16752 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.790513 19508 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:55:42.790916 16950 consensus_peers.cc:485] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 -> Peer aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202:43977): Couldn't send request to peer aa7ec9abef1747e5a9e2404cee265c20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bab1fc6db80c4e53a4f0cacccca74096. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:42.790992 19508 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:42.791339 16883 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.791379 16357 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.791920 19284 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:42.792121 19508 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.795245 16490 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.795890 19520 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:42.795454 19284 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:42.797884 16488 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:42.801419 16067 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bab1fc6db80c4e53a4f0cacccca74096 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:42.802129 16080 catalog_manager.cc:5484] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:42.802492 16223 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:42.804783 16882 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:42.809475 16752 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:42.814170 16620 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:42.819103 16621 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.820134 19508 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:42.822221 16488 consensus_queue.cc:237] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:42.821913 16621 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:42.827895 19543 ts_tablet_manager.cc:902] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316: Initiating tablet copy from peer ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825)
I20240502 01:55:42.833690 16357 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:42.841651 16357 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:42.843104 16883 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.843341 17016 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.843676 19187 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.845542 16752 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.845784 19403 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.847348 19403 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.848518 16357 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.850026 16223 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.850703 19404 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.852444 16621 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:42.853053 19403 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.853861 19404 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.861276 19403 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:42.864686 16754 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:42.868374 16883 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:42.870911 16080 catalog_manager.cc:5484] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } }
I20240502 01:55:42.872761 16357 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:42.873701 17016 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:42.880718 16488 consensus_queue.cc:237] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:42.889254 16357 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.889829 19187 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:42.890678 16754 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.891531 19187 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:42.895754 16883 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.896338 19404 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:42.899938 16620 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.900807 19403 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.901118 16222 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.901975 19403 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:42.902473 16620 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:42.903198 16222 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:42.907362 17016 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:42.908327 19187 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:42.910590 19187 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:42.913995 16223 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:42.914839 16620 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:42.916242 16357 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:42.917434 16067 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:42.921926 16079 catalog_manager.cc:5484] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 reported cstate change: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:42.919461 17016 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:42.922686 16882 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:42.921097 16751 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:42.936328 19543 tablet_copy_client.cc:308] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.197:46825
I20240502 01:55:42.937059 16774 tablet_copy_service.cc:140] P ed2e7e4188d0400b8689e28bbf5096ad: Received BeginTabletCopySession request for tablet dd96134671e1487eba32aaa1dcd6cd67 from peer 0982e15b4c74425cad8d7d642eb63316 ({username='slave'} at 127.31.161.200:49223)
I20240502 01:55:42.937230 16774 tablet_copy_service.cc:161] P ed2e7e4188d0400b8689e28bbf5096ad: Beginning new tablet copy session on tablet dd96134671e1487eba32aaa1dcd6cd67 from peer 0982e15b4c74425cad8d7d642eb63316 at {username='slave'} at 127.31.161.200:49223: session id = 0982e15b4c74425cad8d7d642eb63316-dd96134671e1487eba32aaa1dcd6cd67
W20240502 01:55:42.938973 16424 consensus_peers.cc:485] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 -> Peer aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202:43977): Couldn't send request to peer aa7ec9abef1747e5a9e2404cee265c20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6150a46aa8ca4d9bbd16c141ad2f1a3c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:42.939962 16774 tablet_copy_source_session.cc:205] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:42.941284 19543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd96134671e1487eba32aaa1dcd6cd67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:42.947007 19543 tablet_copy_client.cc:783] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:42.947257 19543 tablet_copy_client.cc:647] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:42.949033 19543 tablet_copy_client.cc:523] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:42.951303 19543 tablet_bootstrap.cc:492] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap starting.
I20240502 01:55:43.001550 19543 tablet_bootstrap.cc:492] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:43.002116 19543 tablet_bootstrap.cc:492] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap complete.
I20240502 01:55:43.002445 19543 ts_tablet_manager.cc:1367] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316: Time spent bootstrapping tablet: real 0.051s user 0.040s sys 0.011s
I20240502 01:55:43.003322 19543 raft_consensus.cc:348] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:43.003695 19543 raft_consensus.cc:729] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0982e15b4c74425cad8d7d642eb63316, State: Initialized, Role: LEARNER
I20240502 01:55:43.003927 19543 consensus_queue.cc:260] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:43.011830 19543 ts_tablet_manager.cc:1398] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316: Time spent starting tablet: real 0.009s user 0.002s sys 0.007s
I20240502 01:55:43.012641 16774 tablet_copy_service.cc:342] P ed2e7e4188d0400b8689e28bbf5096ad: Request end of tablet copy session 0982e15b4c74425cad8d7d642eb63316-dd96134671e1487eba32aaa1dcd6cd67 received from {username='slave'} at 127.31.161.200:49223
I20240502 01:55:43.012795 16774 tablet_copy_service.cc:433] P ed2e7e4188d0400b8689e28bbf5096ad: ending tablet copy session 0982e15b4c74425cad8d7d642eb63316-dd96134671e1487eba32aaa1dcd6cd67 on tablet dd96134671e1487eba32aaa1dcd6cd67 with peer 0982e15b4c74425cad8d7d642eb63316
I20240502 01:55:43.132261 19602 ts_tablet_manager.cc:902] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20: Initiating tablet copy from peer 703f6f043c6041a4b629933690580367 (127.31.161.196:40425)
I20240502 01:55:43.133881 19602 tablet_copy_client.cc:308] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:40425
I20240502 01:55:43.139987 16642 tablet_copy_service.cc:140] P 703f6f043c6041a4b629933690580367: Received BeginTabletCopySession request for tablet f53b466fba6a4655a63a559b0ea56f77 from peer aa7ec9abef1747e5a9e2404cee265c20 ({username='slave'} at 127.31.161.202:49095)
I20240502 01:55:43.140125 16642 tablet_copy_service.cc:161] P 703f6f043c6041a4b629933690580367: Beginning new tablet copy session on tablet f53b466fba6a4655a63a559b0ea56f77 from peer aa7ec9abef1747e5a9e2404cee265c20 at {username='slave'} at 127.31.161.202:49095: session id = aa7ec9abef1747e5a9e2404cee265c20-f53b466fba6a4655a63a559b0ea56f77
I20240502 01:55:43.142056 16642 tablet_copy_source_session.cc:205] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:43.143297 19602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53b466fba6a4655a63a559b0ea56f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:43.147936 19602 tablet_copy_client.cc:783] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:43.148176 19602 tablet_copy_client.cc:647] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:43.149710 19602 tablet_copy_client.cc:523] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:43.152290 19602 tablet_bootstrap.cc:492] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap starting.
I20240502 01:55:43.193522 19602 tablet_bootstrap.cc:492] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:43.193914 19602 tablet_bootstrap.cc:492] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap complete.
I20240502 01:55:43.194219 19602 ts_tablet_manager.cc:1367] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20: Time spent bootstrapping tablet: real 0.042s user 0.037s sys 0.005s
I20240502 01:55:43.194921 19602 raft_consensus.cc:348] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:43.195187 19602 raft_consensus.cc:729] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa7ec9abef1747e5a9e2404cee265c20, State: Initialized, Role: LEARNER
I20240502 01:55:43.195430 19602 consensus_queue.cc:260] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:43.195982 19602 ts_tablet_manager.cc:1398] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:55:43.196717 16642 tablet_copy_service.cc:342] P 703f6f043c6041a4b629933690580367: Request end of tablet copy session aa7ec9abef1747e5a9e2404cee265c20-f53b466fba6a4655a63a559b0ea56f77 received from {username='slave'} at 127.31.161.202:49095
I20240502 01:55:43.196858 16642 tablet_copy_service.cc:433] P 703f6f043c6041a4b629933690580367: ending tablet copy session aa7ec9abef1747e5a9e2404cee265c20-f53b466fba6a4655a63a559b0ea56f77 on tablet f53b466fba6a4655a63a559b0ea56f77 with peer aa7ec9abef1747e5a9e2404cee265c20
I20240502 01:55:43.278952 16751 consensus_queue.cc:237] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:43.281880 17016 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:43.281867 16357 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:43.281965 16882 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:43.281989 16488 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:43.282575 19479 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:43.283079 19615 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:43.283668 19480 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:43.284564 19482 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:43.287256 16223 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:43.287912 16620 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:43.288005 19474 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:43.288645 19474 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:43.292943 19615 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.293638 16620 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.299628 16490 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.303725 16076 catalog_manager.cc:5484] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.306684 16882 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.306990 16223 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.313431 18913 raft_consensus.cc:1203] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:55:43.315351 17014 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.320518 16357 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.326740 16882 consensus_queue.cc:237] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:43.328173 16751 consensus_queue.cc:237] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:43.331574 16222 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 01:55:43.332033 16688 consensus_peers.cc:485] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad -> Peer 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200:38559): Couldn't send request to peer 0982e15b4c74425cad8d7d642eb63316. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a4e2e002a7d441398f3da0ec650b8d2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:43.332545 16883 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:43.332769 16490 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:43.333001 19482 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:43.333360 19474 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:43.333783 16620 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:43.335170 16357 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:43.335891 17015 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:43.335912 19615 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:43.336369 19616 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:43.336696 19615 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:43.337913 16357 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:43.338245 16486 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:43.338701 17015 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:43.338752 19458 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:43.339696 19458 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:43.340044 19629 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:43.340638 19616 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.341897 16223 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:43.342537 19416 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:43.344524 16620 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:43.345189 19416 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:43.353977 16486 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.357337 19602 ts_tablet_manager.cc:902] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20: Initiating tablet copy from peer a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:43.360929 16080 catalog_manager.cc:5484] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.363441 19480 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:43.365754 16358 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.365777 16621 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.371685 17014 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.371737 19416 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.374811 16883 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.375530 16223 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.377527 16358 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.379665 16067 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3a4e2e002a7d441398f3da0ec650b8d2 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:43.380508 16223 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.383533 16486 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.383857 16082 catalog_manager.cc:5484] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "6b9f9aaa1c86420190af75270914449a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.385850 16751 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:43.386502 19458 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:43.409052 16751 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.410477 16882 consensus_queue.cc:237] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:43.415982 16223 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:43.416831 16621 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:43.417448 19458 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:43.418296 19458 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:43.419489 16621 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.421526 16751 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:43.422016 19458 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:43.425441 16358 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:43.426151 16486 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:43.427007 19644 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:43.427439 19458 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:43.429765 19458 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.430941 17014 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:43.430591 16358 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.431650 19416 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:43.432825 17014 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.435248 16068 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 453dff7b204a431592ac15c058db0493 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:43.436841 16081 catalog_manager.cc:5484] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "6b9f9aaa1c86420190af75270914449a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.439071 16751 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.443208 16223 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.436605 17014 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.453356 16486 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.463788 16621 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.476146 16751 consensus_queue.cc:237] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:43.477705 19645 ts_tablet_manager.cc:902] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20: Initiating tablet copy from peer 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:43.484066 16358 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:43.484680 19482 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:43.486371 17015 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:43.486862 16882 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:43.486908 16490 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:43.487483 19480 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:43.487855 16621 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:43.487893 16223 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:43.488456 19615 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:43.489024 19615 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:43.490767 19480 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 01:55:43.495419 19482 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 01:55:43.504745 16819 consensus_peers.cc:485] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a -> Peer 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200:38559): Couldn't send request to peer 0982e15b4c74425cad8d7d642eb63316. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 453dff7b204a431592ac15c058db0493. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:43.511040 19482 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.515146 16358 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.515724 16883 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.517410 16621 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.519941 16486 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.522234 17015 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.523658 16223 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.530009 16082 catalog_manager.cc:5484] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:43.531191 19602 tablet_copy_client.cc:308] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.199:46837
I20240502 01:55:43.532456 17036 tablet_copy_service.cc:140] P a6575ebf978a4631a13ba5c243c3c4f9: Received BeginTabletCopySession request for tablet bab1fc6db80c4e53a4f0cacccca74096 from peer aa7ec9abef1747e5a9e2404cee265c20 ({username='slave'} at 127.31.161.202:59627)
I20240502 01:55:43.532629 17036 tablet_copy_service.cc:161] P a6575ebf978a4631a13ba5c243c3c4f9: Beginning new tablet copy session on tablet bab1fc6db80c4e53a4f0cacccca74096 from peer aa7ec9abef1747e5a9e2404cee265c20 at {username='slave'} at 127.31.161.202:59627: session id = aa7ec9abef1747e5a9e2404cee265c20-bab1fc6db80c4e53a4f0cacccca74096
I20240502 01:55:43.539395 16751 consensus_queue.cc:237] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:43.539935 17036 tablet_copy_source_session.cc:205] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:43.542573 19602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab1fc6db80c4e53a4f0cacccca74096. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:55:43.544363 16688 consensus_peers.cc:485] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad -> Peer 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200:38559): Couldn't send request to peer 0982e15b4c74425cad8d7d642eb63316. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 512eab02225d4544a342811f0a7359aa. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:43.546480 17015 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:43.546553 16222 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:43.547084 19474 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:43.547158 16358 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:43.547859 16883 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:43.547833 19474 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:43.548498 19474 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:43.549068 19474 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:43.550839 19602 tablet_copy_client.cc:783] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:43.551160 19602 tablet_copy_client.cc:647] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:43.552927 19602 tablet_copy_client.cc:523] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:43.554944 16488 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:43.555742 19615 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:43.557057 19602 tablet_bootstrap.cc:492] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap starting.
I20240502 01:55:43.557320 19480 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.561331 16883 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.562944 16620 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:43.563432 16067 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 512eab02225d4544a342811f0a7359aa with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:43.564896 19616 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:43.564860 16222 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.561898 16358 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.567360 16081 catalog_manager.cc:5484] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a reported cstate change: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.569957 16621 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.571475 18780 raft_consensus.cc:1203] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:55:43.572489 16486 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.575870 17014 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:43.603691 19645 tablet_copy_client.cc:308] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:35693
I20240502 01:55:43.623853 16511 tablet_copy_service.cc:140] P 39059854b64045698ded1ba4e8d47304: Received BeginTabletCopySession request for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c from peer aa7ec9abef1747e5a9e2404cee265c20 ({username='slave'} at 127.31.161.202:34163)
I20240502 01:55:43.624028 16511 tablet_copy_service.cc:161] P 39059854b64045698ded1ba4e8d47304: Beginning new tablet copy session on tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c from peer aa7ec9abef1747e5a9e2404cee265c20 at {username='slave'} at 127.31.161.202:34163: session id = aa7ec9abef1747e5a9e2404cee265c20-6150a46aa8ca4d9bbd16c141ad2f1a3c
I20240502 01:55:43.626876 16511 tablet_copy_source_session.cc:205] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:43.628357 19645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:43.638586 19645 tablet_copy_client.cc:783] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:43.645258 19645 tablet_copy_client.cc:647] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:43.646754 19602 tablet_bootstrap.cc:492] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:43.647274 19602 tablet_bootstrap.cc:492] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap complete.
I20240502 01:55:43.647311 19645 tablet_copy_client.cc:523] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:43.647593 19602 ts_tablet_manager.cc:1367] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20: Time spent bootstrapping tablet: real 0.091s user 0.047s sys 0.010s
I20240502 01:55:43.648658 19602 raft_consensus.cc:348] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:43.649044 19602 raft_consensus.cc:729] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa7ec9abef1747e5a9e2404cee265c20, State: Initialized, Role: LEARNER
I20240502 01:55:43.649317 19602 consensus_queue.cc:260] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:43.650058 19602 ts_tablet_manager.cc:1398] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20240502 01:55:43.650965 17036 tablet_copy_service.cc:342] P a6575ebf978a4631a13ba5c243c3c4f9: Request end of tablet copy session aa7ec9abef1747e5a9e2404cee265c20-bab1fc6db80c4e53a4f0cacccca74096 received from {username='slave'} at 127.31.161.202:59627
I20240502 01:55:43.651134 17036 tablet_copy_service.cc:433] P a6575ebf978a4631a13ba5c243c3c4f9: ending tablet copy session aa7ec9abef1747e5a9e2404cee265c20-bab1fc6db80c4e53a4f0cacccca74096 on tablet bab1fc6db80c4e53a4f0cacccca74096 with peer aa7ec9abef1747e5a9e2404cee265c20
I20240502 01:55:43.652575 19645 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap starting.
I20240502 01:55:43.696674 19645 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:43.697147 19645 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap complete.
I20240502 01:55:43.697458 19645 ts_tablet_manager.cc:1367] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20: Time spent bootstrapping tablet: real 0.045s user 0.030s sys 0.014s
I20240502 01:55:43.698266 19645 raft_consensus.cc:348] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:43.698624 19645 raft_consensus.cc:729] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa7ec9abef1747e5a9e2404cee265c20, State: Initialized, Role: LEARNER
I20240502 01:55:43.698856 19645 consensus_queue.cc:260] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:43.699476 19645 ts_tablet_manager.cc:1398] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:55:43.700528 16511 tablet_copy_service.cc:342] P 39059854b64045698ded1ba4e8d47304: Request end of tablet copy session aa7ec9abef1747e5a9e2404cee265c20-6150a46aa8ca4d9bbd16c141ad2f1a3c received from {username='slave'} at 127.31.161.202:34163
I20240502 01:55:43.700673 16511 tablet_copy_service.cc:433] P 39059854b64045698ded1ba4e8d47304: ending tablet copy session aa7ec9abef1747e5a9e2404cee265c20-6150a46aa8ca4d9bbd16c141ad2f1a3c on tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c with peer aa7ec9abef1747e5a9e2404cee265c20
I20240502 01:55:43.748113 18780 raft_consensus.cc:1203] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:55:43.781601 19482 raft_consensus.cc:1050] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad: attempting to promote NON_VOTER 0982e15b4c74425cad8d7d642eb63316 to VOTER
I20240502 01:55:43.782743 19482 consensus_queue.cc:237] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:43.785382 17014 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:43.785701 18913 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:43.786017 19474 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:43.787084 16486 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:43.787142 16222 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:43.787688 16621 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:43.788465 16883 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:43.788647 16357 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:43.788815 19479 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:43.789243 19474 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:43.789484 19616 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:43.789736 19479 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:43.790247 19615 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:43.791973 19482 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:43.800522 18780 raft_consensus.cc:1203] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:55:43.812315 19615 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:43.813918 16882 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:43.818096 17014 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:43.818125 18913 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:43.818809 16081 catalog_manager.cc:5484] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a reported cstate change: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:43.821048 16222 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:43.826122 16486 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:43.833206 16621 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:43.834010 16357 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:43.841163 19713 ts_tablet_manager.cc:902] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316: Initiating tablet copy from peer 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121)
I20240502 01:55:43.841109 16751 consensus_queue.cc:237] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:43.843731 16222 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:43.843791 18913 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:43.843987 16621 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:43.844112 17014 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:43.844390 19479 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:43.844957 19615 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:43.845336 16486 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:43.845571 19482 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:43.845868 19480 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:43.846203 16357 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:43.846597 19615 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:43.847132 19615 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:43.849891 19482 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:43.850481 16486 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:43.851070 16620 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:43.854305 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dd96134671e1487eba32aaa1dcd6cd67 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:43.855207 16222 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:43.855583 17014 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:43.855886 18913 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:43.860849 16357 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:43.861140 16080 catalog_manager.cc:5484] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: config changed from index 6 to 7, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:43.869629 16864 tablet_service.cc:1514] Processing DeleteTablet for tablet dd96134671e1487eba32aaa1dcd6cd67 with delete_type TABLET_DATA_TOMBSTONED (TS 6b9f9aaa1c86420190af75270914449a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49752
I20240502 01:55:43.870373 19718 tablet_replica.cc:330] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a: stopping tablet replica
I20240502 01:55:43.870700 19718 raft_consensus.cc:2229] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:43.871241 19718 raft_consensus.cc:2258] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:43.872277 19718 ts_tablet_manager.cc:1875] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:43.874579 19713 tablet_copy_client.cc:308] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.198:38121
I20240502 01:55:43.878926 19718 ts_tablet_manager.cc:1888] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:43.879045 19718 log.cc:1196] T dd96134671e1487eba32aaa1dcd6cd67 P 6b9f9aaa1c86420190af75270914449a: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal/wals/dd96134671e1487eba32aaa1dcd6cd67
I20240502 01:55:43.879770 16068 catalog_manager.cc:4830] TS 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121): tablet dd96134671e1487eba32aaa1dcd6cd67 (table rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]) successfully deleted
I20240502 01:55:43.881479 16905 tablet_copy_service.cc:140] P 6b9f9aaa1c86420190af75270914449a: Received BeginTabletCopySession request for tablet 453dff7b204a431592ac15c058db0493 from peer 0982e15b4c74425cad8d7d642eb63316 ({username='slave'} at 127.31.161.200:50837)
I20240502 01:55:43.881608 16905 tablet_copy_service.cc:161] P 6b9f9aaa1c86420190af75270914449a: Beginning new tablet copy session on tablet 453dff7b204a431592ac15c058db0493 from peer 0982e15b4c74425cad8d7d642eb63316 at {username='slave'} at 127.31.161.200:50837: session id = 0982e15b4c74425cad8d7d642eb63316-453dff7b204a431592ac15c058db0493
I20240502 01:55:43.883384 16905 tablet_copy_source_session.cc:205] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:43.884574 19713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 453dff7b204a431592ac15c058db0493. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:43.889580 19713 tablet_copy_client.cc:783] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:43.889876 19713 tablet_copy_client.cc:647] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:43.891657 19713 tablet_copy_client.cc:523] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:43.893882 19713 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap starting.
I20240502 01:55:43.912781 19721 ts_tablet_manager.cc:902] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316: Initiating tablet copy from peer ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825)
I20240502 01:55:43.914698 19721 tablet_copy_client.cc:308] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.197:46825
I20240502 01:55:43.915638 16774 tablet_copy_service.cc:140] P ed2e7e4188d0400b8689e28bbf5096ad: Received BeginTabletCopySession request for tablet 3a4e2e002a7d441398f3da0ec650b8d2 from peer 0982e15b4c74425cad8d7d642eb63316 ({username='slave'} at 127.31.161.200:49223)
I20240502 01:55:43.915865 16774 tablet_copy_service.cc:161] P ed2e7e4188d0400b8689e28bbf5096ad: Beginning new tablet copy session on tablet 3a4e2e002a7d441398f3da0ec650b8d2 from peer 0982e15b4c74425cad8d7d642eb63316 at {username='slave'} at 127.31.161.200:49223: session id = 0982e15b4c74425cad8d7d642eb63316-3a4e2e002a7d441398f3da0ec650b8d2
I20240502 01:55:43.918257 16774 tablet_copy_source_session.cc:205] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:43.919492 19721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4e2e002a7d441398f3da0ec650b8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:43.926088 19721 tablet_copy_client.cc:783] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:43.926396 19721 tablet_copy_client.cc:647] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:43.928375 19721 tablet_copy_client.cc:523] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:43.931188 19713 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:43.931566 19713 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap complete.
I20240502 01:55:43.931903 19713 ts_tablet_manager.cc:1367] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316: Time spent bootstrapping tablet: real 0.038s user 0.034s sys 0.004s
I20240502 01:55:43.932049 19721 tablet_bootstrap.cc:492] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap starting.
I20240502 01:55:43.933403 19713 raft_consensus.cc:348] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:43.933784 19713 raft_consensus.cc:729] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0982e15b4c74425cad8d7d642eb63316, State: Initialized, Role: LEARNER
I20240502 01:55:43.934075 19713 consensus_queue.cc:260] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:43.934747 19713 ts_tablet_manager.cc:1398] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 01:55:43.935483 16905 tablet_copy_service.cc:342] P 6b9f9aaa1c86420190af75270914449a: Request end of tablet copy session 0982e15b4c74425cad8d7d642eb63316-453dff7b204a431592ac15c058db0493 received from {username='slave'} at 127.31.161.200:50837
I20240502 01:55:43.935618 16905 tablet_copy_service.cc:433] P 6b9f9aaa1c86420190af75270914449a: ending tablet copy session 0982e15b4c74425cad8d7d642eb63316-453dff7b204a431592ac15c058db0493 on tablet 453dff7b204a431592ac15c058db0493 with peer 0982e15b4c74425cad8d7d642eb63316
I20240502 01:55:43.966233 19713 ts_tablet_manager.cc:902] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316: Initiating tablet copy from peer ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825)
I20240502 01:55:43.970965 19721 tablet_bootstrap.cc:492] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:43.971459 19721 tablet_bootstrap.cc:492] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap complete.
I20240502 01:55:43.971735 19721 ts_tablet_manager.cc:1367] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316: Time spent bootstrapping tablet: real 0.040s user 0.035s sys 0.001s
I20240502 01:55:43.972596 19721 raft_consensus.cc:348] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:43.972921 19721 raft_consensus.cc:729] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0982e15b4c74425cad8d7d642eb63316, State: Initialized, Role: LEARNER
I20240502 01:55:43.973197 19721 consensus_queue.cc:260] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:43.973865 19721 ts_tablet_manager.cc:1398] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:55:43.974624 16774 tablet_copy_service.cc:342] P ed2e7e4188d0400b8689e28bbf5096ad: Request end of tablet copy session 0982e15b4c74425cad8d7d642eb63316-3a4e2e002a7d441398f3da0ec650b8d2 received from {username='slave'} at 127.31.161.200:49223
I20240502 01:55:43.974754 16774 tablet_copy_service.cc:433] P ed2e7e4188d0400b8689e28bbf5096ad: ending tablet copy session 0982e15b4c74425cad8d7d642eb63316-3a4e2e002a7d441398f3da0ec650b8d2 on tablet 3a4e2e002a7d441398f3da0ec650b8d2 with peer 0982e15b4c74425cad8d7d642eb63316
I20240502 01:55:43.997587 19713 tablet_copy_client.cc:308] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.197:46825
I20240502 01:55:43.998333 16774 tablet_copy_service.cc:140] P ed2e7e4188d0400b8689e28bbf5096ad: Received BeginTabletCopySession request for tablet 512eab02225d4544a342811f0a7359aa from peer 0982e15b4c74425cad8d7d642eb63316 ({username='slave'} at 127.31.161.200:49223)
I20240502 01:55:43.998482 16774 tablet_copy_service.cc:161] P ed2e7e4188d0400b8689e28bbf5096ad: Beginning new tablet copy session on tablet 512eab02225d4544a342811f0a7359aa from peer 0982e15b4c74425cad8d7d642eb63316 at {username='slave'} at 127.31.161.200:49223: session id = 0982e15b4c74425cad8d7d642eb63316-512eab02225d4544a342811f0a7359aa
I20240502 01:55:44.001641 16774 tablet_copy_source_session.cc:205] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:44.003065 19713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512eab02225d4544a342811f0a7359aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:44.009738 19713 tablet_copy_client.cc:783] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:44.009975 19713 tablet_copy_client.cc:647] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:44.013182 19713 tablet_copy_client.cc:523] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:44.016950 19713 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316: Bootstrap starting.
I20240502 01:55:44.038467 19169 raft_consensus.cc:1050] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367: attempting to promote NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 to VOTER
I20240502 01:55:44.039742 19169 consensus_queue.cc:237] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:44.042685 18780 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.043385 19355 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.044049 16357 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.044548 16486 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.044653 19516 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.045255 19516 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.050525 16222 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.051534 19516 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.051857 17015 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.052374 19516 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.056448 16751 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.057204 19516 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.058005 16882 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.061530 19355 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.062131 17015 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.065865 19516 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.067081 19713 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:44.067589 19713 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316: Bootstrap complete.
I20240502 01:55:44.067876 19713 ts_tablet_manager.cc:1367] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316: Time spent bootstrapping tablet: real 0.051s user 0.041s sys 0.006s
I20240502 01:55:44.068799 19713 raft_consensus.cc:348] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:44.069103 19713 raft_consensus.cc:729] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0982e15b4c74425cad8d7d642eb63316, State: Initialized, Role: LEARNER
I20240502 01:55:44.069346 19713 consensus_queue.cc:260] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:44.071125 19713 ts_tablet_manager.cc:1398] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20240502 01:55:44.071844 16774 tablet_copy_service.cc:342] P ed2e7e4188d0400b8689e28bbf5096ad: Request end of tablet copy session 0982e15b4c74425cad8d7d642eb63316-512eab02225d4544a342811f0a7359aa received from {username='slave'} at 127.31.161.200:49223
I20240502 01:55:44.071990 16774 tablet_copy_service.cc:433] P ed2e7e4188d0400b8689e28bbf5096ad: ending tablet copy session 0982e15b4c74425cad8d7d642eb63316-512eab02225d4544a342811f0a7359aa on tablet 512eab02225d4544a342811f0a7359aa with peer 0982e15b4c74425cad8d7d642eb63316
I20240502 01:55:44.081039 16883 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.084481 16080 catalog_manager.cc:5484] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.090229 18780 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.100831 16222 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.117736 16751 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.125284 16357 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.125277 16486 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.133716 16621 consensus_queue.cc:237] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:44.137192 16222 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.138721 17015 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.139438 19355 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.139881 18780 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.140460 16883 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.141875 19169 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.142437 19169 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.142736 19516 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.149367 16486 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.150453 19169 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.150700 16883 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.153189 19355 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.153925 16357 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.154451 19355 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.155872 18780 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.156407 16357 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.158605 16486 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.161113 16070 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f53b466fba6a4655a63a559b0ea56f77 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:44.163009 16080 catalog_manager.cc:5484] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a reported cstate change: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.167464 16733 tablet_service.cc:1514] Processing DeleteTablet for tablet f53b466fba6a4655a63a559b0ea56f77 with delete_type TABLET_DATA_TOMBSTONED (TS ed2e7e4188d0400b8689e28bbf5096ad not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47194
I20240502 01:55:44.168357 19775 tablet_replica.cc:330] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad: stopping tablet replica
I20240502 01:55:44.168634 19775 raft_consensus.cc:2229] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:44.169212 19775 raft_consensus.cc:2258] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:44.170178 19775 ts_tablet_manager.cc:1875] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:44.177026 19775 ts_tablet_manager.cc:1888] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:44.177242 19775 log.cc:1196] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal/wals/f53b466fba6a4655a63a559b0ea56f77
I20240502 01:55:44.178174 16067 catalog_manager.cc:4830] TS ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825): tablet f53b466fba6a4655a63a559b0ea56f77 (table rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]) successfully deleted
I20240502 01:55:44.182629 16222 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.182618 17015 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.238171 18913 raft_consensus.cc:1203] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:55:44.267971 19187 raft_consensus.cc:1050] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304: attempting to promote NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 to VOTER
I20240502 01:55:44.268867 19187 consensus_queue.cc:237] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:44.271720 18780 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.272013 17015 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.272228 16621 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.272883 19556 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.273026 16751 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.273208 16883 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.273404 16357 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.273540 19556 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.274379 19781 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.275082 19556 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.275902 19556 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.277761 16222 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.283995 19782 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240502 01:55:44.285522 19781 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.300319 18913 raft_consensus.cc:1203] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:55:44.307296 19783 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.308100 18780 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.312774 16223 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.313194 16621 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.315516 16754 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.316789 16883 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.321744 16080 catalog_manager.cc:5484] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.323971 16357 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.324518 17015 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.337849 16486 consensus_queue.cc:237] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:44.340328 17015 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.340698 19284 raft_consensus.cc:1050] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9: attempting to promote NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 to VOTER
I20240502 01:55:44.340749 18780 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.340981 19556 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.341233 16357 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.341704 19556 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.341809 19284 consensus_queue.cc:237] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:44.341991 19782 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.344332 16223 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.344743 16883 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.345007 19782 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.344702 19187 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.345948 19782 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.345610 17015 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.347494 16883 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.347519 16223 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.350131 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:44.351195 18779 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.351569 16076 catalog_manager.cc:5484] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.353617 16357 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.354933 16621 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.355758 19781 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.358115 16620 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.362705 16733 tablet_service.cc:1514] Processing DeleteTablet for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c with delete_type TABLET_DATA_TOMBSTONED (TS ed2e7e4188d0400b8689e28bbf5096ad not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47194
I20240502 01:55:44.363080 19775 tablet_replica.cc:330] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad: stopping tablet replica
I20240502 01:55:44.363288 19775 raft_consensus.cc:2229] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:44.363569 19775 raft_consensus.cc:2258] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:44.364809 19775 ts_tablet_manager.cc:1875] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:44.371434 18913 raft_consensus.cc:1203] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:55:44.375905 19775 ts_tablet_manager.cc:1888] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:44.376134 19775 log.cc:1196] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal/wals/6150a46aa8ca4d9bbd16c141ad2f1a3c
I20240502 01:55:44.376945 16067 catalog_manager.cc:4830] TS ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825): tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c (table rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]) successfully deleted
I20240502 01:55:44.381731 18779 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.381960 16486 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.382485 16223 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.382730 16751 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.382977 19508 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.383970 19508 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.384500 16357 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.385093 19794 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.385381 19795 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.385723 19508 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.397282 16621 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.397276 16883 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.397899 19284 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.398247 19796 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.399039 19794 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.399601 16486 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.402662 16883 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.405237 16077 catalog_manager.cc:5484] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.407146 18779 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.407402 16357 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.408339 16751 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.422971 16223 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.431485 17015 consensus_queue.cc:237] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:44.427330 16621 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.434273 18779 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.434739 16357 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.434974 16223 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.435478 16751 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.435603 19796 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.435734 16883 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.436170 19794 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.436430 19795 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.436811 19284 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.437295 19508 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.441051 19508 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.441570 16883 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.446292 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bab1fc6db80c4e53a4f0cacccca74096 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:44.445974 16357 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.446913 16223 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.448637 16752 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.450666 18779 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.453568 16076 catalog_manager.cc:5484] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e reported cstate change: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.461930 16602 tablet_service.cc:1514] Processing DeleteTablet for tablet bab1fc6db80c4e53a4f0cacccca74096 with delete_type TABLET_DATA_TOMBSTONED (TS 703f6f043c6041a4b629933690580367 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53048
I20240502 01:55:44.466409 16486 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.467518 19796 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240502 01:55:44.475550 16486 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:44.489542 19810 tablet_replica.cc:330] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367: stopping tablet replica
I20240502 01:55:44.490119 19810 raft_consensus.cc:2229] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:44.490653 19810 raft_consensus.cc:2258] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:44.491603 19810 ts_tablet_manager.cc:1875] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:44.498284 19810 ts_tablet_manager.cc:1888] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:44.498431 19810 log.cc:1196] T bab1fc6db80c4e53a4f0cacccca74096 P 703f6f043c6041a4b629933690580367: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/bab1fc6db80c4e53a4f0cacccca74096
I20240502 01:55:44.499482 16070 catalog_manager.cc:4830] TS 703f6f043c6041a4b629933690580367 (127.31.161.196:40425): tablet bab1fc6db80c4e53a4f0cacccca74096 (table rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]) successfully deleted
I20240502 01:55:44.734688 19416 raft_consensus.cc:1050] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a: attempting to promote NON_VOTER 0982e15b4c74425cad8d7d642eb63316 to VOTER
I20240502 01:55:44.735523 19416 consensus_queue.cc:237] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:44.738757 16486 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.739027 18913 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.739651 17015 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.740226 16357 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.740445 16223 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.740758 19458 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.741330 19840 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.742551 19458 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.743564 19458 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.744441 19840 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.750154 16620 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.750777 16752 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.751210 19842 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.751900 19416 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.751816 19458 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:44.752543 16223 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:44.758105 16357 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:44.758198 16077 catalog_manager.cc:5484] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a reported cstate change: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6b9f9aaa1c86420190af75270914449a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:44.759716 17015 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:44.762233 18913 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:44.762223 16488 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:44.764988 19479 raft_consensus.cc:1050] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad: attempting to promote NON_VOTER 0982e15b4c74425cad8d7d642eb63316 to VOTER
I20240502 01:55:44.766212 19479 consensus_queue.cc:237] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:44.772940 16752 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:44.775991 16488 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.778803 18912 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.780175 16358 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.787020 19615 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.787608 19615 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.787308 16620 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:44.790249 19480 raft_consensus.cc:1050] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad: attempting to promote NON_VOTER 0982e15b4c74425cad8d7d642eb63316 to VOTER
I20240502 01:55:44.791394 19480 consensus_queue.cc:237] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:44.792086 16883 consensus_queue.cc:237] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:44.793550 19615 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.794955 16358 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.795293 16223 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.796073 19842 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.796193 16486 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.796521 19842 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.797225 19840 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.800093 19842 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:44.805315 16068 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 453dff7b204a431592ac15c058db0493 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:44.805732 18911 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.806107 16358 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:44.807348 19841 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.806535 16223 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:44.809055 18911 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:44.814733 16077 catalog_manager.cc:5484] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a reported cstate change: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New cstate: current_term: 1 leader_uuid: "6b9f9aaa1c86420190af75270914449a" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:44.811437 16486 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:44.819423 16733 tablet_service.cc:1514] Processing DeleteTablet for tablet 453dff7b204a431592ac15c058db0493 with delete_type TABLET_DATA_TOMBSTONED (TS ed2e7e4188d0400b8689e28bbf5096ad not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47194
I20240502 01:55:44.819870 19775 tablet_replica.cc:330] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad: stopping tablet replica
I20240502 01:55:44.820081 19775 raft_consensus.cc:2229] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:44.820340 19775 raft_consensus.cc:2258] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:44.826336 17014 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.827369 16883 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.827970 19615 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.828523 19615 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.839566 16223 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.840353 19615 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:44.841626 19775 ts_tablet_manager.cc:1875] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:44.843803 19615 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:44.846712 17015 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.846248 17016 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:44.847277 19842 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.848917 17013 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:44.854532 16488 consensus_queue.cc:237] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:44.854694 16081 catalog_manager.cc:5484] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:44.856858 16620 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:44.860431 19843 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:44.862236 16621 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:44.864233 19775 ts_tablet_manager.cc:1888] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:44.864363 19775 log.cc:1196] T 453dff7b204a431592ac15c058db0493 P ed2e7e4188d0400b8689e28bbf5096ad: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal/wals/453dff7b204a431592ac15c058db0493
I20240502 01:55:44.865177 16067 catalog_manager.cc:4830] TS ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825): tablet 453dff7b204a431592ac15c058db0493 (table rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]) successfully deleted
I20240502 01:55:44.866674 16883 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:44.867228 19783 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:44.872614 17015 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:44.873275 19783 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:44.874683 16752 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:44.875312 19783 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:44.879791 16357 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:44.884327 16358 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:44.884449 16222 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:44.884968 19782 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:44.885550 16224 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:44.886118 19783 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:44.890921 16620 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:44.891587 19783 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:44.900970 19782 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:44.901801 16752 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:44.910109 16359 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:44.911715 16224 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:44.916852 16081 catalog_manager.cc:5484] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:44.925629 16488 consensus_queue.cc:237] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:44.929767 16224 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:44.929860 16359 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 01:55:44.930186 16425 consensus_peers.cc:485] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 -> Peer a7c4573db86a415285d274ca57dde659 (127.31.161.201:44737): Couldn't send request to peer a7c4573db86a415285d274ca57dde659. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 479fcadbde0449f4bfeef0bd2f82ebaf. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:44.930356 19187 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:44.930646 19783 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:44.932533 16752 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:44.934464 19187 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:44.938863 16752 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:44.938148 19875 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:44.947538 18912 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.949654 16488 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.949968 16067 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 479fcadbde0449f4bfeef0bd2f82ebaf with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:44.951124 17016 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.951272 16224 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:44.953866 17015 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:44.954543 19187 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:44.955914 17015 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:44.957952 16077 catalog_manager.cc:5484] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e reported cstate change: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:44.962105 17015 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:44.965462 16619 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:44.966039 16357 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:44.967563 19875 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:44.969796 16882 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:44.970690 16619 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:44.971670 19782 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:44.973335 16358 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.973052 16883 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:44.976912 16883 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:44.977145 16619 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:44.996737 19854 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20240502 01:55:44.997442 16224 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:44.999936 19874 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20240502 01:55:44.999250 19856 log.cc:924] Time spent T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20240502 01:55:45.001636 19867 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240502 01:55:45.011202 19867 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240502 01:55:45.020926 16883 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:45.021891 19854 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:45.026535 16488 consensus_queue.cc:237] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:45.034963 18913 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.036494 17015 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.036626 19874 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:45.037027 19875 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.038455 16882 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.042002 16884 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.042876 16621 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.042892 16358 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.043108 16620 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:45.043514 19187 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.043131 16619 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:45.043944 16222 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.044054 19187 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.044521 19187 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.044684 19880 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:45.045011 19782 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 01:55:45.049441 19479 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:45.050794 16752 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.051456 19782 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.059636 16619 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.072355 19480 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.076284 19782 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:45.077453 18912 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.079031 16077 catalog_manager.cc:5484] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 01:55:45.081331 16687 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.195:35693, user_credentials={real_user=slave}} blocked reactor thread for 81273.7us
I20240502 01:55:45.084249 16075 catalog_manager.cc:5484] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:45.084606 16751 consensus_queue.cc:237] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:45.084307 16884 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.084673 16883 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:45.087133 18912 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:45.087813 19480 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:45.089164 16882 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:45.089578 16223 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:45.090135 19867 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:45.090387 19880 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:45.090234 16486 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.093658 17016 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:45.093998 17013 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:45.095896 16752 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:45.105496 16489 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:45.105484 16357 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:45.107098 16489 consensus_queue.cc:237] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:45.097395 16620 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:45.112344 16223 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.112085 19880 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
W20240502 01:55:45.114159 16425 consensus_peers.cc:485] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 -> Peer a7c4573db86a415285d274ca57dde659 (127.31.161.201:44737): Couldn't send request to peer a7c4573db86a415285d274ca57dde659. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27ce7f50b71e4f478cace3bde6f5e1a3. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:45.114315 19187 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.114020 16223 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.116897 16222 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:45.118175 16884 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.118772 19187 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:45.119526 16358 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.120939 19187 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:45.123710 16081 catalog_manager.cc:5484] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:45.124889 17015 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.124755 18912 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.125510 19782 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:45.125500 16884 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.130949 16752 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.131481 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3a4e2e002a7d441398f3da0ec650b8d2 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:45.133442 16602 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a4e2e002a7d441398f3da0ec650b8d2 with delete_type TABLET_DATA_TOMBSTONED (TS 703f6f043c6041a4b629933690580367 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53048
I20240502 01:55:45.133723 19886 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:45.135097 19782 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:45.135313 19480 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:45.137940 19782 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:45.142109 16752 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:45.145665 16621 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.146449 16357 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:45.150301 19867 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:45.151157 16081 catalog_manager.cc:5484] T 27ce7f50b71e4f478cace3bde6f5e1a3 P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:45.151957 16225 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.153555 17016 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.154294 16067 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 27ce7f50b71e4f478cace3bde6f5e1a3 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:45.155746 16882 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:45.159531 17015 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.163080 19901 tablet_replica.cc:330] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367: stopping tablet replica
I20240502 01:55:45.163357 19901 raft_consensus.cc:2229] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:45.164188 19901 raft_consensus.cc:2258] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:45.163933 16489 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.165263 19901 ts_tablet_manager.cc:1875] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:45.168365 16619 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.169008 19900 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:45.170120 16357 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.172111 16359 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:45.175187 16619 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:45.176777 19901 ts_tablet_manager.cc:1888] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:45.177321 19901 log.cc:1196] T 3a4e2e002a7d441398f3da0ec650b8d2 P 703f6f043c6041a4b629933690580367: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/3a4e2e002a7d441398f3da0ec650b8d2
I20240502 01:55:45.178131 16070 catalog_manager.cc:4830] TS 703f6f043c6041a4b629933690580367 (127.31.161.196:40425): tablet 3a4e2e002a7d441398f3da0ec650b8d2 (table rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]) successfully deleted
I20240502 01:55:45.180577 17013 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:45.149850 16358 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.203236 16225 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:45.205674 16489 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.211134 16752 consensus_queue.cc:237] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:45.215492 18912 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:45.215727 16488 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:45.216116 19867 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:45.217358 16358 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:45.217339 19880 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:45.219307 19881 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:45.221504 16884 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:45.221786 16225 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:45.221902 17013 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:45.222155 19881 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:45.222424 19479 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:45.222887 19867 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:45.224817 19867 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.228760 17013 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.229146 16358 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.230242 16225 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.235323 16884 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.236372 16077 catalog_manager.cc:5484] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.242146 16602 tablet_service.cc:1514] Processing DeleteTablet for tablet 512eab02225d4544a342811f0a7359aa with delete_type TABLET_DATA_TOMBSTONED (TS 703f6f043c6041a4b629933690580367 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53048
I20240502 01:55:45.242601 19901 tablet_replica.cc:330] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367: stopping tablet replica
I20240502 01:55:45.242789 19901 raft_consensus.cc:2229] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:45.243047 19901 raft_consensus.cc:2258] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:45.244700 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 512eab02225d4544a342811f0a7359aa with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:45.245790 18912 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.252208 16489 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:45.255290 19901 ts_tablet_manager.cc:1875] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:45.267956 19901 ts_tablet_manager.cc:1888] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:45.268083 19901 log.cc:1196] T 512eab02225d4544a342811f0a7359aa P 703f6f043c6041a4b629933690580367: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/512eab02225d4544a342811f0a7359aa
I20240502 01:55:45.268846 16070 catalog_manager.cc:4830] TS 703f6f043c6041a4b629933690580367 (127.31.161.196:40425): tablet 512eab02225d4544a342811f0a7359aa (table rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]) successfully deleted
I20240502 01:55:45.272284 16619 consensus_queue.cc:237] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } }
I20240502 01:55:45.274894 16358 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.274966 16489 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.275514 16225 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.276262 19774 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.276381 17015 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.276870 19774 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.277451 19767 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.277792 19903 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.279054 16884 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.280077 16752 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.281947 19903 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.282539 19903 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.305729 19774 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:45.307669 16752 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:45.308636 16884 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:45.310540 17013 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:45.310621 16225 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:45.317210 16077 catalog_manager.cc:5484] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:45.320060 16489 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:45.321496 16358 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } }
I20240502 01:55:45.331806 16619 consensus_queue.cc:237] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:45.335253 16752 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.336074 16358 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.336236 19767 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:45.336489 17015 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.336709 19910 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:45.337280 19767 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 01:55:45.337968 16556 consensus_peers.cc:485] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 -> Peer aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202:43977): Couldn't send request to peer aa7ec9abef1747e5a9e2404cee265c20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc9cc69a319d46b8940f4c7dc2e7e656. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:45.342351 16884 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.342226 19910 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:45.342964 19767 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:45.344923 16884 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:45.344195 17015 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:45.347898 16070 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bc9cc69a319d46b8940f4c7dc2e7e656 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:45.351145 16489 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.351035 16080 catalog_manager.cc:5484] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a reported cstate change: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:45.351853 19903 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:45.352090 16752 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:45.352123 16358 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:45.352703 16225 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.353906 16489 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:45.356103 19923 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:45.358368 16225 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:45.411681 19942 ts_tablet_manager.cc:902] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659: Initiating tablet copy from peer 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:45.413018 19942 tablet_copy_client.cc:308] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:35693
I20240502 01:55:45.413743 16511 tablet_copy_service.cc:140] P 39059854b64045698ded1ba4e8d47304: Received BeginTabletCopySession request for tablet 479fcadbde0449f4bfeef0bd2f82ebaf from peer a7c4573db86a415285d274ca57dde659 ({username='slave'} at 127.31.161.201:59931)
I20240502 01:55:45.413923 16511 tablet_copy_service.cc:161] P 39059854b64045698ded1ba4e8d47304: Beginning new tablet copy session on tablet 479fcadbde0449f4bfeef0bd2f82ebaf from peer a7c4573db86a415285d274ca57dde659 at {username='slave'} at 127.31.161.201:59931: session id = a7c4573db86a415285d274ca57dde659-479fcadbde0449f4bfeef0bd2f82ebaf
I20240502 01:55:45.416297 16511 tablet_copy_source_session.cc:205] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:45.417708 19942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 479fcadbde0449f4bfeef0bd2f82ebaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:45.422236 19942 tablet_copy_client.cc:783] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:45.422518 19942 tablet_copy_client.cc:647] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:45.424190 19942 tablet_copy_client.cc:523] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:45.427069 19942 tablet_bootstrap.cc:492] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659: Bootstrap starting.
I20240502 01:55:45.494787 19942 tablet_bootstrap.cc:492] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:45.495232 19942 tablet_bootstrap.cc:492] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659: Bootstrap complete.
I20240502 01:55:45.495518 19942 ts_tablet_manager.cc:1367] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659: Time spent bootstrapping tablet: real 0.069s user 0.031s sys 0.014s
I20240502 01:55:45.496239 19942 raft_consensus.cc:348] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:45.496541 19942 raft_consensus.cc:729] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a7c4573db86a415285d274ca57dde659, State: Initialized, Role: LEARNER
I20240502 01:55:45.496776 19942 consensus_queue.cc:260] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:45.497629 19942 ts_tablet_manager.cc:1398] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240502 01:55:45.498387 16511 tablet_copy_service.cc:342] P 39059854b64045698ded1ba4e8d47304: Request end of tablet copy session a7c4573db86a415285d274ca57dde659-479fcadbde0449f4bfeef0bd2f82ebaf received from {username='slave'} at 127.31.161.201:59931
I20240502 01:55:45.498512 16511 tablet_copy_service.cc:433] P 39059854b64045698ded1ba4e8d47304: ending tablet copy session a7c4573db86a415285d274ca57dde659-479fcadbde0449f4bfeef0bd2f82ebaf on tablet 479fcadbde0449f4bfeef0bd2f82ebaf with peer a7c4573db86a415285d274ca57dde659
I20240502 01:55:45.509150 19942 ts_tablet_manager.cc:902] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a7c4573db86a415285d274ca57dde659: Initiating tablet copy from peer 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:45.510133 19942 tablet_copy_client.cc:308] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a7c4573db86a415285d274ca57dde659: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:35693
I20240502 01:55:45.510815 16511 tablet_copy_service.cc:140] P 39059854b64045698ded1ba4e8d47304: Received BeginTabletCopySession request for tablet 27ce7f50b71e4f478cace3bde6f5e1a3 from peer a7c4573db86a415285d274ca57dde659 ({username='slave'} at 127.31.161.201:59931)
I20240502 01:55:45.510968 16511 tablet_copy_service.cc:161] P 39059854b64045698ded1ba4e8d47304: Beginning new tablet copy session on tablet 27ce7f50b71e4f478cace3bde6f5e1a3 from peer a7c4573db86a415285d274ca57dde659 at {username='slave'} at 127.31.161.201:59931: session id = a7c4573db86a415285d274ca57dde659-27ce7f50b71e4f478cace3bde6f5e1a3
I20240502 01:55:45.513720 16511 tablet_copy_source_session.cc:205] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:45.514791 19942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27ce7f50b71e4f478cace3bde6f5e1a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:45.518945 19942 tablet_copy_client.cc:783] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:45.519173 19942 tablet_copy_client.cc:647] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:45.520679 19942 tablet_copy_client.cc:523] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a7c4573db86a415285d274ca57dde659: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:45.522855 19942 tablet_bootstrap.cc:492] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a7c4573db86a415285d274ca57dde659: Bootstrap starting.
I20240502 01:55:45.557274 19942 tablet_bootstrap.cc:492] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a7c4573db86a415285d274ca57dde659: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:45.557665 19942 tablet_bootstrap.cc:492] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a7c4573db86a415285d274ca57dde659: Bootstrap complete.
I20240502 01:55:45.557904 19942 ts_tablet_manager.cc:1367] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a7c4573db86a415285d274ca57dde659: Time spent bootstrapping tablet: real 0.035s user 0.032s sys 0.004s
I20240502 01:55:45.558562 19942 raft_consensus.cc:348] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:45.558817 19942 raft_consensus.cc:729] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a7c4573db86a415285d274ca57dde659, State: Initialized, Role: LEARNER
I20240502 01:55:45.559021 19942 consensus_queue.cc:260] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a7c4573db86a415285d274ca57dde659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:45.559566 19942 ts_tablet_manager.cc:1398] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a7c4573db86a415285d274ca57dde659: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:55:45.560277 16511 tablet_copy_service.cc:342] P 39059854b64045698ded1ba4e8d47304: Request end of tablet copy session a7c4573db86a415285d274ca57dde659-27ce7f50b71e4f478cace3bde6f5e1a3 received from {username='slave'} at 127.31.161.201:59931
I20240502 01:55:45.560438 16511 tablet_copy_service.cc:433] P 39059854b64045698ded1ba4e8d47304: ending tablet copy session a7c4573db86a415285d274ca57dde659-27ce7f50b71e4f478cace3bde6f5e1a3 on tablet 27ce7f50b71e4f478cace3bde6f5e1a3 with peer a7c4573db86a415285d274ca57dde659
I20240502 01:55:45.687516 17015 consensus_queue.cc:237] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:45.690491 16358 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.690569 16884 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.691036 19796 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.691176 16619 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.691340 19794 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.692364 16225 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.692406 16752 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.692951 19796 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.693504 19796 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.693832 16489 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.695905 19796 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.696342 19794 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.696630 19796 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:45.717227 16752 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:45.723590 16076 catalog_manager.cc:5484] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:45.723806 16884 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:45.726492 16225 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:45.751256 16489 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
W20240502 01:55:45.762317 16952 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.196:40425, user_credentials={real_user=slave}} blocked reactor thread for 66044.4us
I20240502 01:55:45.763455 17015 consensus_queue.cc:237] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:45.766593 16358 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:45.769598 16619 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:45.773577 16884 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.773769 16752 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.774645 16489 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.774943 19796 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:45.775265 19979 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:45.775933 19795 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:45.778237 16225 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.779014 19796 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:45.782049 19979 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:45.783278 16225 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:45.784849 16621 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.787734 16067 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a8f7e3c8b6d6434db2a99d7139599a5d with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:45.787343 16884 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:45.788205 16357 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.789891 16752 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:45.792141 16077 catalog_manager.cc:5484] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:45.798228 19795 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:45.798799 19795 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:45.799839 16489 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:45.804162 19988 ts_tablet_manager.cc:902] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20: Initiating tablet copy from peer 703f6f043c6041a4b629933690580367 (127.31.161.196:40425)
I20240502 01:55:45.805922 16357 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:45.812203 16619 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
W20240502 01:55:45.816748 16951 consensus_peers.cc:485] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 -> Peer a7c4573db86a415285d274ca57dde659 (127.31.161.201:44737): Couldn't send request to peer a7c4573db86a415285d274ca57dde659. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8f7e3c8b6d6434db2a99d7139599a5d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:45.823114 19988 tablet_copy_client.cc:308] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:40425
I20240502 01:55:45.823781 16642 tablet_copy_service.cc:140] P 703f6f043c6041a4b629933690580367: Received BeginTabletCopySession request for tablet bc9cc69a319d46b8940f4c7dc2e7e656 from peer aa7ec9abef1747e5a9e2404cee265c20 ({username='slave'} at 127.31.161.202:49095)
I20240502 01:55:45.823937 16642 tablet_copy_service.cc:161] P 703f6f043c6041a4b629933690580367: Beginning new tablet copy session on tablet bc9cc69a319d46b8940f4c7dc2e7e656 from peer aa7ec9abef1747e5a9e2404cee265c20 at {username='slave'} at 127.31.161.202:49095: session id = aa7ec9abef1747e5a9e2404cee265c20-bc9cc69a319d46b8940f4c7dc2e7e656
I20240502 01:55:45.824198 16489 consensus_queue.cc:237] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } }
I20240502 01:55:45.826367 16642 tablet_copy_source_session.cc:205] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:45.827383 17013 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.827682 16884 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.828222 19900 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.828439 16619 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.828979 16357 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.829238 16225 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.829506 19782 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.829805 19900 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.830317 19900 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.830489 16752 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 01:55:45.832520 19782 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.833034 19783 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.833734 19988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9cc69a319d46b8940f4c7dc2e7e656. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:45.845438 19988 tablet_copy_client.cc:783] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:45.845773 19988 tablet_copy_client.cc:647] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:45.847404 19988 tablet_copy_client.cc:523] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:45.850944 19988 tablet_bootstrap.cc:492] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap starting.
I20240502 01:55:45.874317 19783 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:45.881273 16621 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:45.884703 16752 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:45.891618 16077 catalog_manager.cc:5484] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:45.890152 17013 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:45.900266 16358 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:45.901010 16489 consensus_queue.cc:237] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:45.907817 16359 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.908421 19994 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:45.909797 16752 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 01:55:45.909844 16424 consensus_peers.cc:485] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 -> Peer 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200:38559): Couldn't send request to peer 0982e15b4c74425cad8d7d642eb63316. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fa6224f34ef4bc58797f6cf801ff6da. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:45.909937 16621 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.910538 16884 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.910655 19782 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:45.911105 19782 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.913704 19900 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:45.915272 16884 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:45.917179 19994 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:45.917712 16359 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:45.918862 17015 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.920256 19782 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 01:55:45.922725 17015 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:45.924958 16752 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:45.925969 16067 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1fa6224f34ef4bc58797f6cf801ff6da with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 01:55:45.929806 16621 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:45.931902 16080 catalog_manager.cc:5484] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:45.938277 16884 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:45.940831 18647 raft_consensus.cc:1203] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:55:45.941743 16222 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 01:55:45.943727 19900 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 01:55:45.945690 16222 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } }
I20240502 01:55:45.957240 19988 tablet_bootstrap.cc:492] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:45.957686 19988 tablet_bootstrap.cc:492] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap complete.
I20240502 01:55:45.958011 19988 ts_tablet_manager.cc:1367] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20: Time spent bootstrapping tablet: real 0.107s user 0.055s sys 0.007s
I20240502 01:55:45.958364 16222 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:45.958868 19988 raft_consensus.cc:348] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:45.959192 19988 raft_consensus.cc:729] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa7ec9abef1747e5a9e2404cee265c20, State: Initialized, Role: LEARNER
I20240502 01:55:45.959424 19988 consensus_queue.cc:260] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:45.961594 19988 ts_tablet_manager.cc:1398] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 01:55:45.962707 16642 tablet_copy_service.cc:342] P 703f6f043c6041a4b629933690580367: Request end of tablet copy session aa7ec9abef1747e5a9e2404cee265c20-bc9cc69a319d46b8940f4c7dc2e7e656 received from {username='slave'} at 127.31.161.202:49095
I20240502 01:55:45.962862 16642 tablet_copy_service.cc:433] P 703f6f043c6041a4b629933690580367: ending tablet copy session aa7ec9abef1747e5a9e2404cee265c20-bc9cc69a319d46b8940f4c7dc2e7e656 on tablet bc9cc69a319d46b8940f4c7dc2e7e656 with peer aa7ec9abef1747e5a9e2404cee265c20
I20240502 01:55:45.987426 18647 raft_consensus.cc:1203] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:55:46.213786 20029 ts_tablet_manager.cc:902] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659: Initiating tablet copy from peer a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:46.214946 20029 tablet_copy_client.cc:308] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.199:46837
I20240502 01:55:46.221177 17036 tablet_copy_service.cc:140] P a6575ebf978a4631a13ba5c243c3c4f9: Received BeginTabletCopySession request for tablet a8f7e3c8b6d6434db2a99d7139599a5d from peer a7c4573db86a415285d274ca57dde659 ({username='slave'} at 127.31.161.201:32955)
I20240502 01:55:46.221335 17036 tablet_copy_service.cc:161] P a6575ebf978a4631a13ba5c243c3c4f9: Beginning new tablet copy session on tablet a8f7e3c8b6d6434db2a99d7139599a5d from peer a7c4573db86a415285d274ca57dde659 at {username='slave'} at 127.31.161.201:32955: session id = a7c4573db86a415285d274ca57dde659-a8f7e3c8b6d6434db2a99d7139599a5d
I20240502 01:55:46.223414 17036 tablet_copy_source_session.cc:205] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:46.224889 20029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8f7e3c8b6d6434db2a99d7139599a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:46.229442 20029 tablet_copy_client.cc:783] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:46.229646 20029 tablet_copy_client.cc:647] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:46.231282 20029 tablet_copy_client.cc:523] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:46.235903 20029 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659: Bootstrap starting.
I20240502 01:55:46.257727 18779 raft_consensus.cc:1203] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:55:46.275532 20029 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:46.276005 20029 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659: Bootstrap complete.
I20240502 01:55:46.276321 20029 ts_tablet_manager.cc:1367] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659: Time spent bootstrapping tablet: real 0.041s user 0.037s sys 0.000s
I20240502 01:55:46.277215 20029 raft_consensus.cc:348] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:46.277557 20029 raft_consensus.cc:729] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a7c4573db86a415285d274ca57dde659, State: Initialized, Role: LEARNER
I20240502 01:55:46.277791 20029 consensus_queue.cc:260] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:46.278396 20029 ts_tablet_manager.cc:1398] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:55:46.279147 17036 tablet_copy_service.cc:342] P a6575ebf978a4631a13ba5c243c3c4f9: Request end of tablet copy session a7c4573db86a415285d274ca57dde659-a8f7e3c8b6d6434db2a99d7139599a5d received from {username='slave'} at 127.31.161.201:32955
I20240502 01:55:46.279284 17036 tablet_copy_service.cc:433] P a6575ebf978a4631a13ba5c243c3c4f9: ending tablet copy session a7c4573db86a415285d274ca57dde659-a8f7e3c8b6d6434db2a99d7139599a5d on tablet a8f7e3c8b6d6434db2a99d7139599a5d with peer a7c4573db86a415285d274ca57dde659
I20240502 01:55:46.387161 20063 ts_tablet_manager.cc:902] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316: Initiating tablet copy from peer 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:46.389743 20063 tablet_copy_client.cc:308] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:35693
I20240502 01:55:46.390549 16511 tablet_copy_service.cc:140] P 39059854b64045698ded1ba4e8d47304: Received BeginTabletCopySession request for tablet 1fa6224f34ef4bc58797f6cf801ff6da from peer 0982e15b4c74425cad8d7d642eb63316 ({username='slave'} at 127.31.161.200:41079)
I20240502 01:55:46.390722 16511 tablet_copy_service.cc:161] P 39059854b64045698ded1ba4e8d47304: Beginning new tablet copy session on tablet 1fa6224f34ef4bc58797f6cf801ff6da from peer 0982e15b4c74425cad8d7d642eb63316 at {username='slave'} at 127.31.161.200:41079: session id = 0982e15b4c74425cad8d7d642eb63316-1fa6224f34ef4bc58797f6cf801ff6da
I20240502 01:55:46.392668 16511 tablet_copy_source_session.cc:205] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:46.393855 20063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa6224f34ef4bc58797f6cf801ff6da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:46.399580 20063 tablet_copy_client.cc:783] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:46.399823 20063 tablet_copy_client.cc:647] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:46.402200 20063 tablet_copy_client.cc:523] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:46.404589 20063 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316: Bootstrap starting.
I20240502 01:55:46.419634 19994 raft_consensus.cc:1050] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304: attempting to promote NON_VOTER a7c4573db86a415285d274ca57dde659 to VOTER
I20240502 01:55:46.420467 19994 consensus_queue.cc:237] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:46.423255 18647 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:46.423285 17013 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:46.423867 19782 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:46.424808 16884 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:46.425595 19783 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:46.426044 16222 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:46.427253 19900 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:46.427444 16752 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:46.429421 16357 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:46.429975 19782 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:46.430562 19900 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:46.430995 16621 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:46.433138 19782 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:46.433799 19782 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:46.444350 20063 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:46.444804 20063 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316: Bootstrap complete.
I20240502 01:55:46.445154 20063 ts_tablet_manager.cc:1367] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316: Time spent bootstrapping tablet: real 0.041s user 0.032s sys 0.008s
I20240502 01:55:46.445989 20063 raft_consensus.cc:348] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:46.446321 20063 raft_consensus.cc:729] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0982e15b4c74425cad8d7d642eb63316, State: Initialized, Role: LEARNER
I20240502 01:55:46.446559 20063 consensus_queue.cc:260] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:46.453814 20063 ts_tablet_manager.cc:1398] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20240502 01:55:46.453529 19782 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.454664 16511 tablet_copy_service.cc:342] P 39059854b64045698ded1ba4e8d47304: Request end of tablet copy session 0982e15b4c74425cad8d7d642eb63316-1fa6224f34ef4bc58797f6cf801ff6da received from {username='slave'} at 127.31.161.200:41079
I20240502 01:55:46.454828 16511 tablet_copy_service.cc:433] P 39059854b64045698ded1ba4e8d47304: ending tablet copy session 0982e15b4c74425cad8d7d642eb63316-1fa6224f34ef4bc58797f6cf801ff6da on tablet 1fa6224f34ef4bc58797f6cf801ff6da with peer 0982e15b4c74425cad8d7d642eb63316
I20240502 01:55:46.459072 16884 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.460202 16752 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.461794 16357 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.466177 16080 catalog_manager.cc:5484] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:46.466449 16222 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.469818 18647 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.491796 16619 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.497416 17015 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.500711 19782 raft_consensus.cc:1050] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304: attempting to promote NON_VOTER a7c4573db86a415285d274ca57dde659 to VOTER
I20240502 01:55:46.501973 19782 consensus_queue.cc:237] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:46.512972 18647 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:46.514449 19783 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:46.518810 16489 consensus_queue.cc:237] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:46.513396 17013 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:46.519948 19783 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:46.522487 16225 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:46.523104 19994 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:46.524725 16359 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:46.524888 18647 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:46.525640 19994 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:46.526492 19994 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:46.527858 16884 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:46.528357 19994 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:46.528733 16621 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:46.529290 19994 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:46.529119 19900 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.533254 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 479fcadbde0449f4bfeef0bd2f82ebaf with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:46.533816 18647 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.534111 16359 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.538614 16882 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:46.531147 16619 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.539157 19994 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:46.543882 16081 catalog_manager.cc:5484] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.544318 16884 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.549587 16733 tablet_service.cc:1514] Processing DeleteTablet for tablet 479fcadbde0449f4bfeef0bd2f82ebaf with delete_type TABLET_DATA_TOMBSTONED (TS ed2e7e4188d0400b8689e28bbf5096ad not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47194
I20240502 01:55:46.551558 16620 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:46.552186 19994 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:46.559433 20090 tablet_replica.cc:330] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad: stopping tablet replica
I20240502 01:55:46.558784 16225 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.560267 20090 raft_consensus.cc:2229] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:46.561735 20090 raft_consensus.cc:2258] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:46.563397 20090 ts_tablet_manager.cc:1875] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:46.565683 16359 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:46.566792 16224 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:46.567416 19900 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:46.568027 19900 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:46.585830 20088 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.587632 18647 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.591784 20090 ts_tablet_manager.cc:1888] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:46.591922 20090 log.cc:1196] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal/wals/479fcadbde0449f4bfeef0bd2f82ebaf
I20240502 01:55:46.592782 16067 catalog_manager.cc:4830] TS ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825): tablet 479fcadbde0449f4bfeef0bd2f82ebaf (table rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]) successfully deleted
I20240502 01:55:46.593262 17013 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:46.593112 16081 catalog_manager.cc:5484] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a7c4573db86a415285d274ca57dde659 reported cstate change: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.594307 16223 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.595369 16752 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:46.600692 19782 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:46.602469 19782 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:46.602648 17013 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.603454 17015 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.604421 16882 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.611171 16752 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.612752 16357 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.614145 16620 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:46.671974 19903 raft_consensus.cc:1050] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367: attempting to promote NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 to VOTER
I20240502 01:55:46.673151 19903 consensus_queue.cc:237] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:46.676424 16488 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:46.677120 16223 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:46.678000 20120 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:46.678515 17013 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:46.678755 18780 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:46.678803 20120 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:46.679781 16357 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:46.679881 19910 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:46.680406 19910 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:46.680918 19910 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:46.685271 16882 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:46.686389 16752 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:46.686986 19923 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:46.688652 18647 raft_consensus.cc:1203] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:55:46.688956 19923 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:46.724912 19910 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:46.730194 18780 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:46.730830 16751 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:46.733603 16223 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:46.736330 16082 catalog_manager.cc:5484] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 reported cstate change: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:46.744781 16357 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:46.747478 16884 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:46.759405 16488 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:46.759569 17013 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:46.762320 16620 consensus_queue.cc:237] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:46.767261 16225 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:46.767400 18779 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:46.767796 19910 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:46.768085 16486 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:46.768100 20121 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:46.771014 19903 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:46.771384 16882 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:46.771934 19903 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:46.773365 17015 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:46.773767 16357 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:46.774449 19903 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:46.775868 19923 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:46.776093 19910 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:46.776366 16225 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:46.778251 16359 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:46.779752 16486 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:46.780339 16070 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bc9cc69a319d46b8940f4c7dc2e7e656 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:46.782253 17013 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:46.783941 18779 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:46.784346 16083 catalog_manager.cc:5484] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:46.788100 16882 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:46.791026 16733 tablet_service.cc:1514] Processing DeleteTablet for tablet bc9cc69a319d46b8940f4c7dc2e7e656 with delete_type TABLET_DATA_TOMBSTONED (TS ed2e7e4188d0400b8689e28bbf5096ad not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:47194
I20240502 01:55:46.791837 20090 tablet_replica.cc:330] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad: stopping tablet replica
I20240502 01:55:46.792026 20090 raft_consensus.cc:2229] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:46.792222 20090 raft_consensus.cc:2258] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:46.793380 20090 ts_tablet_manager.cc:1875] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:46.798382 20090 ts_tablet_manager.cc:1888] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:46.798547 20090 log.cc:1196] T bc9cc69a319d46b8940f4c7dc2e7e656 P ed2e7e4188d0400b8689e28bbf5096ad: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal/wals/bc9cc69a319d46b8940f4c7dc2e7e656
I20240502 01:55:46.799324 16067 catalog_manager.cc:4830] TS ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825): tablet bc9cc69a319d46b8940f4c7dc2e7e656 (table rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]) successfully deleted
I20240502 01:55:46.837152 18912 raft_consensus.cc:1203] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 01:55:47.019326 16488 consensus_queue.cc:237] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:47.022318 18647 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:47.022636 16752 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:47.023003 19994 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:47.023357 16357 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:47.023408 19900 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:47.023905 19994 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:47.024135 16882 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:47.024722 19994 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:47.025123 16225 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:47.026830 17013 raft_consensus.cc:1261] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:47.027339 19994 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:47.029064 19994 consensus_queue.cc:1035] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:47.029426 19782 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.030058 18647 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.031162 17013 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.033977 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 27ce7f50b71e4f478cace3bde6f5e1a3 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:47.034503 16357 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.035213 16225 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.035260 16882 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.038017 16752 raft_consensus.cc:2932] T 27ce7f50b71e4f478cace3bde6f5e1a3 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.041229 16079 catalog_manager.cc:5484] T 27ce7f50b71e4f478cace3bde6f5e1a3 P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.045945 16602 tablet_service.cc:1514] Processing DeleteTablet for tablet 27ce7f50b71e4f478cace3bde6f5e1a3 with delete_type TABLET_DATA_TOMBSTONED (TS 703f6f043c6041a4b629933690580367 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53048
I20240502 01:55:47.046561 20149 tablet_replica.cc:330] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367: stopping tablet replica
I20240502 01:55:47.046833 20149 raft_consensus.cc:2229] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:47.047243 20149 raft_consensus.cc:2258] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:47.048133 20149 ts_tablet_manager.cc:1875] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:47.054518 20149 ts_tablet_manager.cc:1888] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:47.054656 20149 log.cc:1196] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 703f6f043c6041a4b629933690580367: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/27ce7f50b71e4f478cace3bde6f5e1a3
I20240502 01:55:47.055424 16070 catalog_manager.cc:4830] TS 703f6f043c6041a4b629933690580367 (127.31.161.196:40425): tablet 27ce7f50b71e4f478cace3bde6f5e1a3 (table rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]) successfully deleted
I20240502 01:55:47.126201 19974 raft_consensus.cc:1050] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9: attempting to promote NON_VOTER a7c4573db86a415285d274ca57dde659 to VOTER
I20240502 01:55:47.127086 19974 consensus_queue.cc:237] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:47.129978 16752 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:47.130201 16882 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:47.130692 20062 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:47.131364 20062 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:47.132491 18647 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:47.133127 16357 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:47.133970 16620 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:47.134044 20153 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:47.134537 20062 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:47.134903 20152 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:47.136276 16488 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:47.136881 19978 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:47.141505 20153 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.142676 16225 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:47.142242 16620 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.144626 20062 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:47.146201 16357 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.146473 16882 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.146502 16752 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.151037 16076 catalog_manager.cc:5484] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:47.153668 16225 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.158692 18647 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.159065 16486 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.441300 19783 raft_consensus.cc:1050] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304: attempting to promote NON_VOTER 0982e15b4c74425cad8d7d642eb63316 to VOTER
I20240502 01:55:47.442497 19783 consensus_queue.cc:237] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:47.446060 17013 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:47.446194 16752 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:47.446686 19994 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:47.446810 16620 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:47.447005 16223 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:47.447413 19994 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:47.447758 19900 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:47.448064 16882 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:47.448062 19782 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:47.448112 18912 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:47.448666 19900 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:47.449182 16357 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 01:55:47.449214 19900 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:47.449890 19782 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 01:55:47.462320 19994 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:47.463016 16357 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:47.470338 16076 catalog_manager.cc:5484] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:47.470494 17013 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:47.471333 16225 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:47.471755 16752 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:47.474339 16882 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:47.496474 16620 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:47.505304 18912 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:47.508011 16486 consensus_queue.cc:237] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:47.511534 16225 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:47.511987 16752 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:47.512254 20088 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:47.512732 19900 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:47.514986 16357 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:47.516038 20088 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:47.518621 17013 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:47.519209 19994 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:47.519716 19994 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:47.528470 17013 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:47.541239 18913 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:47.542586 16882 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:47.542553 16357 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:47.545878 16752 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:47.547688 19900 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:47.548511 19783 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:47.549741 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1fa6224f34ef4bc58797f6cf801ff6da with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:47.549456 16882 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:47.551240 16076 catalog_manager.cc:5484] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:47.551957 16225 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:47.556075 18913 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:47.559629 16602 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fa6224f34ef4bc58797f6cf801ff6da with delete_type TABLET_DATA_TOMBSTONED (TS 703f6f043c6041a4b629933690580367 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53048
I20240502 01:55:47.562662 17015 consensus_queue.cc:237] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:47.568483 18647 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:47.569365 16882 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:47.569540 16486 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:47.569978 20153 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:47.570557 19978 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:47.570804 20153 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:47.571286 16357 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:47.573248 19978 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:47.575992 16752 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:47.576730 20154 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:47.577391 16882 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.581066 16751 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.582286 16225 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 01:55:47.582769 19978 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 01:55:47.582959 20153 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.584270 16223 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.590822 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a8f7e3c8b6d6434db2a99d7139599a5d with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:47.591589 16486 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.591753 18647 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.589536 16077 catalog_manager.cc:5484] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.595645 16357 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:47.601648 20189 tablet_replica.cc:330] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367: stopping tablet replica
I20240502 01:55:47.601949 20189 raft_consensus.cc:2229] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:47.602471 20189 raft_consensus.cc:2258] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:47.604334 20189 ts_tablet_manager.cc:1875] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:47.606561 16602 tablet_service.cc:1514] Processing DeleteTablet for tablet a8f7e3c8b6d6434db2a99d7139599a5d with delete_type TABLET_DATA_TOMBSTONED (TS 703f6f043c6041a4b629933690580367 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:53048
I20240502 01:55:47.620719 20189 ts_tablet_manager.cc:1888] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:47.620846 20189 log.cc:1196] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/1fa6224f34ef4bc58797f6cf801ff6da
I20240502 01:55:47.622089 16070 catalog_manager.cc:4830] TS 703f6f043c6041a4b629933690580367 (127.31.161.196:40425): tablet 1fa6224f34ef4bc58797f6cf801ff6da (table rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]) successfully deleted
I20240502 01:55:47.622503 20189 tablet_replica.cc:330] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367: stopping tablet replica
I20240502 01:55:47.622686 20189 raft_consensus.cc:2229] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:47.623023 20189 raft_consensus.cc:2258] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:47.624294 20189 ts_tablet_manager.cc:1875] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:47.631027 20189 ts_tablet_manager.cc:1888] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 01:55:47.631179 20189 log.cc:1196] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/a8f7e3c8b6d6434db2a99d7139599a5d
I20240502 01:55:47.631881 16070 catalog_manager.cc:4830] TS 703f6f043c6041a4b629933690580367 (127.31.161.196:40425): tablet a8f7e3c8b6d6434db2a99d7139599a5d (table rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]) successfully deleted
I20240502 01:55:47.972769 16471 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:55:47.980305 16864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:47.997558 18893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.015973 16602 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:48.049347 18760 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.054107 18626 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.074819 16204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.121147 16995 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:48.143503 16733 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:48.215903 16339 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.402966 16471 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 01:55:48.428189 16995 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:48.433740 18893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.439728 16204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.450147 16602 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:48.453816 16864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:48.469918 18760 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.486469 18626 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.540292 16733 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:48.544000 16339 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:48.675335 16488 consensus_queue.cc:237] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
W20240502 01:55:48.678377 16424 consensus_peers.cc:485] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 -> Peer 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200:38559): Couldn't send request to peer 0982e15b4c74425cad8d7d642eb63316. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 48a81a0b7d47479e929d80041f34a3b1. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:48.678340 16358 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.679306 18647 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.679301 20287 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.679872 19782 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.681046 16225 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.681627 20287 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.681977 16620 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.683396 20287 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.686640 20086 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:48.689285 16620 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:48.691150 18647 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:48.696749 16752 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.696766 17015 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.697399 19782 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.697345 16358 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:48.701129 16081 catalog_manager.cc:5484] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 reported cstate change: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:48.702442 16225 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:48.703060 20288 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.712407 17015 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:48.731688 16752 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:48.755106 16488 consensus_queue.cc:237] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:48.758551 18779 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.758716 17015 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 01:55:48.759263 16425 consensus_peers.cc:485] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 -> Peer a7c4573db86a415285d274ca57dde659 (127.31.161.201:44737): Couldn't send request to peer a7c4573db86a415285d274ca57dde659. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6150a46aa8ca4d9bbd16c141ad2f1a3c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:48.759603 20086 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.760121 16882 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.760371 20086 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.760861 20288 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.760929 16620 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.761716 20288 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.768780 16358 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.769347 19782 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.769676 16225 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.771068 20287 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.770843 19782 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:48.772897 16225 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:48.773636 16882 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:48.775435 16358 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:48.777561 18779 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:48.779285 17013 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:48.782454 16079 catalog_manager.cc:5484] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a reported cstate change: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:48.786585 16620 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:48.816606 16488 consensus_queue.cc:237] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:48.819530 18913 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.819875 16358 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.820436 19782 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 01:55:48.820811 16424 consensus_peers.cc:485] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 -> Peer aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202:43977): Couldn't send request to peer aa7ec9abef1747e5a9e2404cee265c20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 18e676bcd8b04f18a6814d6ebbd1b488. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:48.820802 20288 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.822041 17013 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.822361 16882 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.822541 19782 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.822954 20086 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.823174 16752 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.823876 20086 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.829239 16225 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.829795 20086 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.839406 20323 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:48.839956 16752 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:48.841399 17015 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:48.842592 16358 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:48.846172 16225 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:48.847321 16083 catalog_manager.cc:5484] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:48.849357 18913 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:48.853722 16882 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:48.876827 17015 consensus_queue.cc:237] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:48.880014 18647 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.880697 20239 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.882413 16752 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.883862 20239 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.886266 16488 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.886308 16358 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.887063 20239 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.887388 20153 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.887914 16225 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.888796 20153 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.893659 16357 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:48.897212 16882 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.897809 20153 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:48.904358 16081 catalog_manager.cc:5484] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:48.904492 18647 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:48.905691 20346 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.905735 16752 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:48.918183 16882 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:48.922716 16223 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
W20240502 01:55:48.924276 16950 consensus_peers.cc:485] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 -> Peer 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200:38559): Couldn't send request to peer 0982e15b4c74425cad8d7d642eb63316. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8f7e3c8b6d6434db2a99d7139599a5d. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:48.927423 16488 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:48.946220 16752 consensus_queue.cc:237] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:48.949031 16488 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.949183 18913 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.950076 19881 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.950618 20148 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.951470 16358 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.952049 20330 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.966164 16223 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.966838 20330 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.970494 16882 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:48.971894 20148 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.973806 20358 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:48.975644 17015 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 01:55:48.979736 16689 consensus_peers.cc:485] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad -> Peer a7c4573db86a415285d274ca57dde659 (127.31.161.201:44737): Couldn't send request to peer a7c4573db86a415285d274ca57dde659. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 512eab02225d4544a342811f0a7359aa. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:48.980230 16488 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:48.981520 19881 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:48.984925 17015 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:48.986459 16358 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:48.988557 16081 catalog_manager.cc:5484] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:48.998265 16223 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:49.007493 18913 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:49.013301 16884 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:49.034180 16752 consensus_queue.cc:237] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:49.037449 17015 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 01:55:49.037853 16688 consensus_peers.cc:485] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad -> Peer aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202:43977): Couldn't send request to peer aa7ec9abef1747e5a9e2404cee265c20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd96134671e1487eba32aaa1dcd6cd67. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:49.037883 16488 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.037994 19881 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.038553 16223 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.038715 19881 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.038934 16620 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.039291 19881 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.041667 19881 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.045814 16358 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.046391 18913 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.046339 19881 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.047061 20357 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240502 01:55:49.052459 19881 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:49.053263 16358 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:49.056706 16486 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:49.060235 16081 catalog_manager.cc:5484] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:49.062978 16620 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:49.067162 18912 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:49.067579 17015 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:49.083693 16223 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:49.103667 16620 consensus_queue.cc:237] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:49.107260 16486 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.107302 18647 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.108230 19910 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.108613 20120 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.124989 16358 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.125949 16223 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.126725 20120 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.128170 19910 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.137017 16884 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.138077 19910 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 01:55:49.146700 16556 consensus_peers.cc:485] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 -> Peer 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200:38559): Couldn't send request to peer 0982e15b4c74425cad8d7d642eb63316. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6cde24f949334006aa9e241e844e81eb. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:49.147545 17015 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.148744 20389 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.166702 20392 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.167685 16357 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.171468 16884 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.178562 16081 catalog_manager.cc:5484] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a reported cstate change: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.185918 16225 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.199724 17015 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.204917 20399 ts_tablet_manager.cc:902] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316: Initiating tablet copy from peer 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:49.213191 18645 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.217363 20402 ts_tablet_manager.cc:902] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659: Initiating tablet copy from peer 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:49.222648 16486 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.238673 20402 tablet_copy_client.cc:308] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:35693
I20240502 01:55:49.239300 16511 tablet_copy_service.cc:140] P 39059854b64045698ded1ba4e8d47304: Received BeginTabletCopySession request for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c from peer a7c4573db86a415285d274ca57dde659 ({username='slave'} at 127.31.161.201:59931)
I20240502 01:55:49.239445 16511 tablet_copy_service.cc:161] P 39059854b64045698ded1ba4e8d47304: Beginning new tablet copy session on tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c from peer a7c4573db86a415285d274ca57dde659 at {username='slave'} at 127.31.161.201:59931: session id = a7c4573db86a415285d274ca57dde659-6150a46aa8ca4d9bbd16c141ad2f1a3c
I20240502 01:55:49.242503 16511 tablet_copy_source_session.cc:205] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:49.244571 20402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:49.252099 20402 tablet_copy_client.cc:783] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:49.252338 20402 tablet_copy_client.cc:647] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:49.254185 20402 tablet_copy_client.cc:523] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:49.257881 20402 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659: Bootstrap starting.
I20240502 01:55:49.272822 16884 consensus_queue.cc:237] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
W20240502 01:55:49.278795 16820 consensus_peers.cc:485] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a -> Peer a7c4573db86a415285d274ca57dde659 (127.31.161.201:44737): Couldn't send request to peer a7c4573db86a415285d274ca57dde659. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 453dff7b204a431592ac15c058db0493. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:49.279884 18912 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.280438 16486 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.280658 20173 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.283530 16225 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.296229 20418 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.297598 20173 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.305928 16358 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.307837 20418 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240502 01:55:49.318245 20402 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:49.318718 20402 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659: Bootstrap complete.
I20240502 01:55:49.319078 20402 ts_tablet_manager.cc:1367] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659: Time spent bootstrapping tablet: real 0.061s user 0.050s sys 0.009s
I20240502 01:55:49.319921 20402 raft_consensus.cc:348] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:49.320349 20402 raft_consensus.cc:729] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a7c4573db86a415285d274ca57dde659, State: Initialized, Role: LEARNER
I20240502 01:55:49.320570 20402 consensus_queue.cc:260] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:49.321352 20402 ts_tablet_manager.cc:1398] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:49.322666 16511 tablet_copy_service.cc:342] P 39059854b64045698ded1ba4e8d47304: Request end of tablet copy session a7c4573db86a415285d274ca57dde659-6150a46aa8ca4d9bbd16c141ad2f1a3c received from {username='slave'} at 127.31.161.201:59931
I20240502 01:55:49.322805 16511 tablet_copy_service.cc:433] P 39059854b64045698ded1ba4e8d47304: ending tablet copy session a7c4573db86a415285d274ca57dde659-6150a46aa8ca4d9bbd16c141ad2f1a3c on tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c with peer a7c4573db86a415285d274ca57dde659
I20240502 01:55:49.327729 17015 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.328440 20420 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.328631 16620 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.332154 20420 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.338858 20173 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:49.339337 16620 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:49.344554 18912 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:49.346086 16081 catalog_manager.cc:5484] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 reported cstate change: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "6b9f9aaa1c86420190af75270914449a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:49.347607 16225 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:49.354324 16358 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:49.361953 20414 ts_tablet_manager.cc:902] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316: Initiating tablet copy from peer a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:49.362433 20413 ts_tablet_manager.cc:902] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20: Initiating tablet copy from peer 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:49.362957 16488 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:49.365799 17015 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:49.371909 20399 tablet_copy_client.cc:308] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:35693
I20240502 01:55:49.373176 16511 tablet_copy_service.cc:140] P 39059854b64045698ded1ba4e8d47304: Received BeginTabletCopySession request for tablet 48a81a0b7d47479e929d80041f34a3b1 from peer 0982e15b4c74425cad8d7d642eb63316 ({username='slave'} at 127.31.161.200:41079)
I20240502 01:55:49.373370 16511 tablet_copy_service.cc:161] P 39059854b64045698ded1ba4e8d47304: Beginning new tablet copy session on tablet 48a81a0b7d47479e929d80041f34a3b1 from peer 0982e15b4c74425cad8d7d642eb63316 at {username='slave'} at 127.31.161.200:41079: session id = 0982e15b4c74425cad8d7d642eb63316-48a81a0b7d47479e929d80041f34a3b1
I20240502 01:55:49.376433 16511 tablet_copy_source_session.cc:205] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:49.378031 20399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a81a0b7d47479e929d80041f34a3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:49.384272 20414 tablet_copy_client.cc:308] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.199:46837
I20240502 01:55:49.385108 20399 tablet_copy_client.cc:783] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:49.385423 20399 tablet_copy_client.cc:647] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:49.388409 20413 tablet_copy_client.cc:308] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:35693
I20240502 01:55:49.389264 16511 tablet_copy_service.cc:140] P 39059854b64045698ded1ba4e8d47304: Received BeginTabletCopySession request for tablet 18e676bcd8b04f18a6814d6ebbd1b488 from peer aa7ec9abef1747e5a9e2404cee265c20 ({username='slave'} at 127.31.161.202:34163)
I20240502 01:55:49.389423 16511 tablet_copy_service.cc:161] P 39059854b64045698ded1ba4e8d47304: Beginning new tablet copy session on tablet 18e676bcd8b04f18a6814d6ebbd1b488 from peer aa7ec9abef1747e5a9e2404cee265c20 at {username='slave'} at 127.31.161.202:34163: session id = aa7ec9abef1747e5a9e2404cee265c20-18e676bcd8b04f18a6814d6ebbd1b488
I20240502 01:55:49.393865 16511 tablet_copy_source_session.cc:205] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:49.396963 20413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18e676bcd8b04f18a6814d6ebbd1b488. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:49.397797 16486 consensus_queue.cc:237] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
W20240502 01:55:49.403503 16424 consensus_peers.cc:485] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 -> Peer aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202:43977): Couldn't send request to peer aa7ec9abef1747e5a9e2404cee265c20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fa6224f34ef4bc58797f6cf801ff6da. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:49.403586 20413 tablet_copy_client.cc:783] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:49.403861 20413 tablet_copy_client.cc:647] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:49.403987 16884 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.404125 16225 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.404659 16752 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.404731 20323 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.405526 18913 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.405826 17015 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.406446 20086 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.406926 20315 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.407668 20413 tablet_copy_client.cc:523] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:49.411293 20413 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap starting.
I20240502 01:55:49.411921 20086 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.414096 20323 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.414583 17036 tablet_copy_service.cc:140] P a6575ebf978a4631a13ba5c243c3c4f9: Received BeginTabletCopySession request for tablet a8f7e3c8b6d6434db2a99d7139599a5d from peer 0982e15b4c74425cad8d7d642eb63316 ({username='slave'} at 127.31.161.200:60783)
I20240502 01:55:49.414800 17036 tablet_copy_service.cc:161] P a6575ebf978a4631a13ba5c243c3c4f9: Beginning new tablet copy session on tablet a8f7e3c8b6d6434db2a99d7139599a5d from peer 0982e15b4c74425cad8d7d642eb63316 at {username='slave'} at 127.31.161.200:60783: session id = 0982e15b4c74425cad8d7d642eb63316-a8f7e3c8b6d6434db2a99d7139599a5d
I20240502 01:55:49.417692 17036 tablet_copy_source_session.cc:205] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:49.419598 20414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8f7e3c8b6d6434db2a99d7139599a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:49.426461 20414 tablet_copy_client.cc:783] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:49.426723 20414 tablet_copy_client.cc:647] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:49.428450 20414 tablet_copy_client.cc:523] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:49.431948 20414 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316: Bootstrap starting.
I20240502 01:55:49.436726 16358 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.438688 20323 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.443162 20402 ts_tablet_manager.cc:902] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659: Initiating tablet copy from peer ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825)
I20240502 01:55:49.447782 20323 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:49.452261 20399 tablet_copy_client.cc:523] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:49.451977 16882 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:49.455783 20399 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap starting.
I20240502 01:55:49.460876 16081 catalog_manager.cc:5484] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a reported cstate change: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:49.463749 18913 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:49.471434 17015 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:49.475253 16358 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:49.496765 16751 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:49.504506 20413 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:49.505153 20413 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap complete.
I20240502 01:55:49.505645 20413 ts_tablet_manager.cc:1367] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20: Time spent bootstrapping tablet: real 0.094s user 0.042s sys 0.016s
I20240502 01:55:49.506690 20413 raft_consensus.cc:348] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:49.507200 20413 raft_consensus.cc:729] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa7ec9abef1747e5a9e2404cee265c20, State: Initialized, Role: LEARNER
I20240502 01:55:49.507607 20413 consensus_queue.cc:260] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:49.510522 20455 ts_tablet_manager.cc:902] T dd96134671e1487eba32aaa1dcd6cd67 P aa7ec9abef1747e5a9e2404cee265c20: Initiating tablet copy from peer ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825)
I20240502 01:55:49.522329 16223 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:49.531893 20456 ts_tablet_manager.cc:902] T 6cde24f949334006aa9e241e844e81eb P 0982e15b4c74425cad8d7d642eb63316: Initiating tablet copy from peer 703f6f043c6041a4b629933690580367 (127.31.161.196:40425)
I20240502 01:55:49.533159 20455 tablet_copy_client.cc:308] T dd96134671e1487eba32aaa1dcd6cd67 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.197:46825
I20240502 01:55:49.533241 20402 tablet_copy_client.cc:308] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.197:46825
I20240502 01:55:49.544896 16774 tablet_copy_service.cc:140] P ed2e7e4188d0400b8689e28bbf5096ad: Received BeginTabletCopySession request for tablet dd96134671e1487eba32aaa1dcd6cd67 from peer aa7ec9abef1747e5a9e2404cee265c20 ({username='slave'} at 127.31.161.202:42599)
I20240502 01:55:49.545135 20413 ts_tablet_manager.cc:1398] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20: Time spent starting tablet: real 0.039s user 0.011s sys 0.024s
I20240502 01:55:49.545120 16774 tablet_copy_service.cc:161] P ed2e7e4188d0400b8689e28bbf5096ad: Beginning new tablet copy session on tablet dd96134671e1487eba32aaa1dcd6cd67 from peer aa7ec9abef1747e5a9e2404cee265c20 at {username='slave'} at 127.31.161.202:42599: session id = aa7ec9abef1747e5a9e2404cee265c20-dd96134671e1487eba32aaa1dcd6cd67
I20240502 01:55:49.547246 16774 tablet_copy_source_session.cc:205] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:49.549257 16511 tablet_copy_service.cc:342] P 39059854b64045698ded1ba4e8d47304: Request end of tablet copy session aa7ec9abef1747e5a9e2404cee265c20-18e676bcd8b04f18a6814d6ebbd1b488 received from {username='slave'} at 127.31.161.202:34163
I20240502 01:55:49.549417 16511 tablet_copy_service.cc:433] P 39059854b64045698ded1ba4e8d47304: ending tablet copy session aa7ec9abef1747e5a9e2404cee265c20-18e676bcd8b04f18a6814d6ebbd1b488 on tablet 18e676bcd8b04f18a6814d6ebbd1b488 with peer aa7ec9abef1747e5a9e2404cee265c20
I20240502 01:55:49.551245 20455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd96134671e1487eba32aaa1dcd6cd67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:49.557538 20455 tablet_copy_client.cc:783] T dd96134671e1487eba32aaa1dcd6cd67 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:49.557775 20455 tablet_copy_client.cc:647] T dd96134671e1487eba32aaa1dcd6cd67 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:49.559376 20455 tablet_copy_client.cc:523] T dd96134671e1487eba32aaa1dcd6cd67 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:49.563915 20455 tablet_bootstrap.cc:492] T dd96134671e1487eba32aaa1dcd6cd67 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap starting.
I20240502 01:55:49.581403 16774 tablet_copy_service.cc:140] P ed2e7e4188d0400b8689e28bbf5096ad: Received BeginTabletCopySession request for tablet 512eab02225d4544a342811f0a7359aa from peer a7c4573db86a415285d274ca57dde659 ({username='slave'} at 127.31.161.201:44943)
I20240502 01:55:49.581584 16774 tablet_copy_service.cc:161] P ed2e7e4188d0400b8689e28bbf5096ad: Beginning new tablet copy session on tablet 512eab02225d4544a342811f0a7359aa from peer a7c4573db86a415285d274ca57dde659 at {username='slave'} at 127.31.161.201:44943: session id = a7c4573db86a415285d274ca57dde659-512eab02225d4544a342811f0a7359aa
I20240502 01:55:49.590965 16774 tablet_copy_source_session.cc:205] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:49.592494 20402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512eab02225d4544a342811f0a7359aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:49.597574 16620 consensus_queue.cc:237] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
W20240502 01:55:49.606937 16556 consensus_peers.cc:485] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 -> Peer 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200:38559): Couldn't send request to peer 0982e15b4c74425cad8d7d642eb63316. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cec84c633b0a4c4fb6e074c79432fe61. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:49.607518 16884 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.608659 18645 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.609230 20392 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.610343 19910 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.612836 17015 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.613878 16358 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.613919 20391 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.614554 20391 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.616086 16223 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.616694 20389 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.621459 16488 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.634969 20389 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.654953 20414 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:49.655416 20414 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316: Bootstrap complete.
I20240502 01:55:49.655738 20414 ts_tablet_manager.cc:1367] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316: Time spent bootstrapping tablet: real 0.224s user 0.061s sys 0.027s
I20240502 01:55:49.656566 20414 raft_consensus.cc:348] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:49.656908 20414 raft_consensus.cc:729] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0982e15b4c74425cad8d7d642eb63316, State: Initialized, Role: LEARNER
I20240502 01:55:49.661100 20456 tablet_copy_client.cc:308] T 6cde24f949334006aa9e241e844e81eb P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:40425
I20240502 01:55:49.664273 20414 consensus_queue.cc:260] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:49.666501 20414 ts_tablet_manager.cc:1398] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240502 01:55:49.668126 17036 tablet_copy_service.cc:342] P a6575ebf978a4631a13ba5c243c3c4f9: Request end of tablet copy session 0982e15b4c74425cad8d7d642eb63316-a8f7e3c8b6d6434db2a99d7139599a5d received from {username='slave'} at 127.31.161.200:60783
I20240502 01:55:49.668280 17036 tablet_copy_service.cc:433] P a6575ebf978a4631a13ba5c243c3c4f9: ending tablet copy session 0982e15b4c74425cad8d7d642eb63316-a8f7e3c8b6d6434db2a99d7139599a5d on tablet a8f7e3c8b6d6434db2a99d7139599a5d with peer 0982e15b4c74425cad8d7d642eb63316
I20240502 01:55:49.678416 16642 tablet_copy_service.cc:140] P 703f6f043c6041a4b629933690580367: Received BeginTabletCopySession request for tablet 6cde24f949334006aa9e241e844e81eb from peer 0982e15b4c74425cad8d7d642eb63316 ({username='slave'} at 127.31.161.200:36469)
I20240502 01:55:49.678579 16642 tablet_copy_service.cc:161] P 703f6f043c6041a4b629933690580367: Beginning new tablet copy session on tablet 6cde24f949334006aa9e241e844e81eb from peer 0982e15b4c74425cad8d7d642eb63316 at {username='slave'} at 127.31.161.200:36469: session id = 0982e15b4c74425cad8d7d642eb63316-6cde24f949334006aa9e241e844e81eb
I20240502 01:55:49.678301 20389 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.680790 16357 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.682008 16642 tablet_copy_source_session.cc:205] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:49.685456 20456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cde24f949334006aa9e241e844e81eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:49.688416 16081 catalog_manager.cc:5484] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.690337 20455 tablet_bootstrap.cc:492] T dd96134671e1487eba32aaa1dcd6cd67 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:49.690677 18913 raft_consensus.cc:1203] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 01:55:49.690780 20455 tablet_bootstrap.cc:492] T dd96134671e1487eba32aaa1dcd6cd67 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap complete.
I20240502 01:55:49.691076 20455 ts_tablet_manager.cc:1367] T dd96134671e1487eba32aaa1dcd6cd67 P aa7ec9abef1747e5a9e2404cee265c20: Time spent bootstrapping tablet: real 0.127s user 0.049s sys 0.007s
I20240502 01:55:49.691892 20455 raft_consensus.cc:348] T dd96134671e1487eba32aaa1dcd6cd67 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:49.692263 20455 raft_consensus.cc:729] T dd96134671e1487eba32aaa1dcd6cd67 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa7ec9abef1747e5a9e2404cee265c20, State: Initialized, Role: LEARNER
I20240502 01:55:49.694108 20456 tablet_copy_client.cc:783] T 6cde24f949334006aa9e241e844e81eb P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:49.694917 20456 tablet_copy_client.cc:647] T 6cde24f949334006aa9e241e844e81eb P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:49.697571 20456 tablet_copy_client.cc:523] T 6cde24f949334006aa9e241e844e81eb P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:49.701236 20456 tablet_bootstrap.cc:492] T 6cde24f949334006aa9e241e844e81eb P 0982e15b4c74425cad8d7d642eb63316: Bootstrap starting.
I20240502 01:55:49.704073 16488 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.692497 20455 consensus_queue.cc:260] T dd96134671e1487eba32aaa1dcd6cd67 P aa7ec9abef1747e5a9e2404cee265c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:49.713048 20455 ts_tablet_manager.cc:1398] T dd96134671e1487eba32aaa1dcd6cd67 P aa7ec9abef1747e5a9e2404cee265c20: Time spent starting tablet: real 0.022s user 0.004s sys 0.000s
I20240502 01:55:49.714310 16774 tablet_copy_service.cc:342] P ed2e7e4188d0400b8689e28bbf5096ad: Request end of tablet copy session aa7ec9abef1747e5a9e2404cee265c20-dd96134671e1487eba32aaa1dcd6cd67 received from {username='slave'} at 127.31.161.202:42599
I20240502 01:55:49.714533 16774 tablet_copy_service.cc:433] P ed2e7e4188d0400b8689e28bbf5096ad: ending tablet copy session aa7ec9abef1747e5a9e2404cee265c20-dd96134671e1487eba32aaa1dcd6cd67 on tablet dd96134671e1487eba32aaa1dcd6cd67 with peer aa7ec9abef1747e5a9e2404cee265c20
I20240502 01:55:49.724735 20402 tablet_copy_client.cc:783] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:49.725162 20402 tablet_copy_client.cc:647] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:49.732568 18645 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.733417 16223 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.734381 20402 tablet_copy_client.cc:523] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:49.735127 16884 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.745476 17013 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.756408 20402 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659: Bootstrap starting.
I20240502 01:55:49.758188 20456 tablet_bootstrap.cc:492] T 6cde24f949334006aa9e241e844e81eb P 0982e15b4c74425cad8d7d642eb63316: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:49.762573 20399 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:49.785480 20456 tablet_bootstrap.cc:492] T 6cde24f949334006aa9e241e844e81eb P 0982e15b4c74425cad8d7d642eb63316: Bootstrap complete.
I20240502 01:55:49.785502 20399 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap complete.
I20240502 01:55:49.785858 20456 ts_tablet_manager.cc:1367] T 6cde24f949334006aa9e241e844e81eb P 0982e15b4c74425cad8d7d642eb63316: Time spent bootstrapping tablet: real 0.085s user 0.044s sys 0.011s
I20240502 01:55:49.786654 20456 raft_consensus.cc:348] T 6cde24f949334006aa9e241e844e81eb P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:49.786994 20456 raft_consensus.cc:729] T 6cde24f949334006aa9e241e844e81eb P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0982e15b4c74425cad8d7d642eb63316, State: Initialized, Role: LEARNER
I20240502 01:55:49.787197 20456 consensus_queue.cc:260] T 6cde24f949334006aa9e241e844e81eb P 0982e15b4c74425cad8d7d642eb63316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:49.787848 20456 ts_tablet_manager.cc:1398] T 6cde24f949334006aa9e241e844e81eb P 0982e15b4c74425cad8d7d642eb63316: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 01:55:49.788758 18780 raft_consensus.cc:1203] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 01:55:49.789050 16642 tablet_copy_service.cc:342] P 703f6f043c6041a4b629933690580367: Request end of tablet copy session 0982e15b4c74425cad8d7d642eb63316-6cde24f949334006aa9e241e844e81eb received from {username='slave'} at 127.31.161.200:36469
I20240502 01:55:49.789212 16642 tablet_copy_service.cc:433] P 703f6f043c6041a4b629933690580367: ending tablet copy session 0982e15b4c74425cad8d7d642eb63316-6cde24f949334006aa9e241e844e81eb on tablet 6cde24f949334006aa9e241e844e81eb with peer 0982e15b4c74425cad8d7d642eb63316
I20240502 01:55:49.789916 20399 ts_tablet_manager.cc:1367] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316: Time spent bootstrapping tablet: real 0.330s user 0.045s sys 0.012s
I20240502 01:55:49.790652 20399 raft_consensus.cc:348] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:49.790972 20399 raft_consensus.cc:729] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0982e15b4c74425cad8d7d642eb63316, State: Initialized, Role: LEARNER
I20240502 01:55:49.791160 20399 consensus_queue.cc:260] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:49.791724 20399 ts_tablet_manager.cc:1398] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 01:55:49.793917 16511 tablet_copy_service.cc:342] P 39059854b64045698ded1ba4e8d47304: Request end of tablet copy session 0982e15b4c74425cad8d7d642eb63316-48a81a0b7d47479e929d80041f34a3b1 received from {username='slave'} at 127.31.161.200:41079
I20240502 01:55:49.794071 16511 tablet_copy_service.cc:433] P 39059854b64045698ded1ba4e8d47304: ending tablet copy session 0982e15b4c74425cad8d7d642eb63316-48a81a0b7d47479e929d80041f34a3b1 on tablet 48a81a0b7d47479e929d80041f34a3b1 with peer 0982e15b4c74425cad8d7d642eb63316
I20240502 01:55:49.802330 16752 consensus_queue.cc:237] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
W20240502 01:55:49.806382 16689 consensus_peers.cc:485] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad -> Peer a7c4573db86a415285d274ca57dde659 (127.31.161.201:44737): Couldn't send request to peer a7c4573db86a415285d274ca57dde659. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b55b98b06aca47988b14f486973cc370. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:49.806530 17015 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.806630 16620 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.806994 18913 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.807572 20358 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.808068 19881 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.808768 20358 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.810617 16488 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.812290 20476 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.820801 16223 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.821004 18645 raft_consensus.cc:1203] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 01:55:49.821805 20357 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.829465 16358 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.829341 20402 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:49.830210 20476 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.830655 20402 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659: Bootstrap complete.
I20240502 01:55:49.830937 20402 ts_tablet_manager.cc:1367] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659: Time spent bootstrapping tablet: real 0.075s user 0.039s sys 0.009s
I20240502 01:55:49.831748 20402 raft_consensus.cc:348] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:49.832053 20402 raft_consensus.cc:729] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a7c4573db86a415285d274ca57dde659, State: Initialized, Role: LEARNER
I20240502 01:55:49.832300 20402 consensus_queue.cc:260] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:49.835750 20402 ts_tablet_manager.cc:1398] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 01:55:49.836385 16774 tablet_copy_service.cc:342] P ed2e7e4188d0400b8689e28bbf5096ad: Request end of tablet copy session a7c4573db86a415285d274ca57dde659-512eab02225d4544a342811f0a7359aa received from {username='slave'} at 127.31.161.201:44943
I20240502 01:55:49.836530 16774 tablet_copy_service.cc:433] P ed2e7e4188d0400b8689e28bbf5096ad: ending tablet copy session a7c4573db86a415285d274ca57dde659-512eab02225d4544a342811f0a7359aa on tablet 512eab02225d4544a342811f0a7359aa with peer a7c4573db86a415285d274ca57dde659
I20240502 01:55:49.842967 20476 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:49.843605 16358 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:49.846491 16621 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:49.847997 18913 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:49.849900 17015 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:49.853049 16076 catalog_manager.cc:5484] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:49.859019 16488 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:49.859375 16223 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:49.878273 16621 consensus_queue.cc:237] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:49.881181 18780 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.881847 16488 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.882130 17015 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 01:55:49.882231 16556 consensus_peers.cc:485] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 -> Peer 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200:38559): Couldn't send request to peer 0982e15b4c74425cad8d7d642eb63316. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc9cc69a319d46b8940f4c7dc2e7e656. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:49.882208 16223 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.882407 20387 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.882725 20392 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.883320 16884 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.883658 20387 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.884106 20391 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.884690 20387 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.892397 16358 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.893020 20387 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.902858 20402 ts_tablet_manager.cc:902] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659: Initiating tablet copy from peer 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121)
I20240502 01:55:49.903489 20391 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.906282 16357 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.907085 16489 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.913575 16081 catalog_manager.cc:5484] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.915935 18780 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.917423 16225 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.922314 16884 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.941006 17013 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:49.942557 18780 raft_consensus.cc:1203] T dd96134671e1487eba32aaa1dcd6cd67 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 01:55:49.948515 20402 tablet_copy_client.cc:308] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.198:38121
I20240502 01:55:49.949353 16905 tablet_copy_service.cc:140] P 6b9f9aaa1c86420190af75270914449a: Received BeginTabletCopySession request for tablet 453dff7b204a431592ac15c058db0493 from peer a7c4573db86a415285d274ca57dde659 ({username='slave'} at 127.31.161.201:37183)
I20240502 01:55:49.949520 16905 tablet_copy_service.cc:161] P 6b9f9aaa1c86420190af75270914449a: Beginning new tablet copy session on tablet 453dff7b204a431592ac15c058db0493 from peer a7c4573db86a415285d274ca57dde659 at {username='slave'} at 127.31.161.201:37183: session id = a7c4573db86a415285d274ca57dde659-453dff7b204a431592ac15c058db0493
I20240502 01:55:49.952126 16905 tablet_copy_source_session.cc:205] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:49.953871 20402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 453dff7b204a431592ac15c058db0493. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:49.957615 17013 consensus_queue.cc:237] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:49.958915 20402 tablet_copy_client.cc:783] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:49.959153 20402 tablet_copy_client.cc:647] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:49.960923 18780 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 01:55:49.961036 16951 consensus_peers.cc:485] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 -> Peer a7c4573db86a415285d274ca57dde659 (127.31.161.201:44737): Couldn't send request to peer a7c4573db86a415285d274ca57dde659. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 297561b66dd7472ab817c2a23723565c. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:49.961254 20402 tablet_copy_client.cc:523] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:49.961526 20346 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.962792 16488 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.962808 16621 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.962960 16884 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:49.963357 20347 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.963919 20347 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.964479 20402 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659: Bootstrap starting.
I20240502 01:55:49.964470 20347 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:49.996311 20153 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:50.003212 16358 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:50.002955 18780 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:50.003073 16621 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:50.004688 16488 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:50.012117 16081 catalog_manager.cc:5484] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:50.017531 20347 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:50.019435 16223 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:50.026839 20455 ts_tablet_manager.cc:902] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20: Initiating tablet copy from peer 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:50.028142 20455 tablet_copy_client.cc:308] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:35693
I20240502 01:55:50.028519 16358 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:50.031334 16511 tablet_copy_service.cc:140] P 39059854b64045698ded1ba4e8d47304: Received BeginTabletCopySession request for tablet 1fa6224f34ef4bc58797f6cf801ff6da from peer aa7ec9abef1747e5a9e2404cee265c20 ({username='slave'} at 127.31.161.202:34163)
I20240502 01:55:50.031539 16511 tablet_copy_service.cc:161] P 39059854b64045698ded1ba4e8d47304: Beginning new tablet copy session on tablet 1fa6224f34ef4bc58797f6cf801ff6da from peer aa7ec9abef1747e5a9e2404cee265c20 at {username='slave'} at 127.31.161.202:34163: session id = aa7ec9abef1747e5a9e2404cee265c20-1fa6224f34ef4bc58797f6cf801ff6da
I20240502 01:55:50.033439 20346 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:50.034513 16511 tablet_copy_source_session.cc:205] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:50.039084 20455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa6224f34ef4bc58797f6cf801ff6da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:50.039870 16884 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:50.051885 20455 tablet_copy_client.cc:783] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:50.052248 20455 tablet_copy_client.cc:647] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:50.052845 20402 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:50.053303 20402 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659: Bootstrap complete.
I20240502 01:55:50.053594 20402 ts_tablet_manager.cc:1367] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659: Time spent bootstrapping tablet: real 0.089s user 0.051s sys 0.025s
I20240502 01:55:50.054010 20455 tablet_copy_client.cc:523] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:50.054361 20402 raft_consensus.cc:348] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:50.054695 20402 raft_consensus.cc:729] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a7c4573db86a415285d274ca57dde659, State: Initialized, Role: LEARNER
I20240502 01:55:50.054898 20402 consensus_queue.cc:260] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:50.055527 20402 ts_tablet_manager.cc:1398] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:55:50.056524 16905 tablet_copy_service.cc:342] P 6b9f9aaa1c86420190af75270914449a: Request end of tablet copy session a7c4573db86a415285d274ca57dde659-453dff7b204a431592ac15c058db0493 received from {username='slave'} at 127.31.161.201:37183
I20240502 01:55:50.056674 16905 tablet_copy_service.cc:433] P 6b9f9aaa1c86420190af75270914449a: ending tablet copy session a7c4573db86a415285d274ca57dde659-453dff7b204a431592ac15c058db0493 on tablet 453dff7b204a431592ac15c058db0493 with peer a7c4573db86a415285d274ca57dde659
I20240502 01:55:50.061636 20455 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap starting.
I20240502 01:55:50.062356 16223 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:50.065398 20347 raft_consensus.cc:1050] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9: attempting to promote NON_VOTER 0982e15b4c74425cad8d7d642eb63316 to VOTER
I20240502 01:55:50.066711 20347 consensus_queue.cc:237] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:50.071132 18913 raft_consensus.cc:1203] T 6cde24f949334006aa9e241e844e81eb P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 01:55:50.075398 18645 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.075565 18913 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.076195 20153 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.076718 20526 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.078140 16488 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.078298 16884 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.078742 20153 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.079981 20532 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.081135 16752 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.081954 20153 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.104251 20399 ts_tablet_manager.cc:902] T cec84c633b0a4c4fb6e074c79432fe61 P 0982e15b4c74425cad8d7d642eb63316: Initiating tablet copy from peer 703f6f043c6041a4b629933690580367 (127.31.161.196:40425)
I20240502 01:55:50.126111 16358 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.126827 20153 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.127516 16225 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.128347 20549 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.136425 20532 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.137233 16488 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.144349 16082 catalog_manager.cc:5484] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.144347 18647 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.151394 16884 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.157939 18913 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.159006 16752 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.160439 16225 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.183511 16358 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.188264 17013 consensus_queue.cc:237] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:50.193814 18913 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.194891 18647 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.195497 20549 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.196605 20549 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.200803 20455 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:50.201370 20455 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap complete.
I20240502 01:55:50.202312 20455 ts_tablet_manager.cc:1367] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20: Time spent bootstrapping tablet: real 0.141s user 0.044s sys 0.013s
I20240502 01:55:50.202740 16752 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.203198 20455 raft_consensus.cc:348] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:50.203495 20455 raft_consensus.cc:729] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa7ec9abef1747e5a9e2404cee265c20, State: Initialized, Role: LEARNER
I20240502 01:55:50.203759 20455 consensus_queue.cc:260] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:50.204054 16884 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.204406 20347 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.204851 20347 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.206146 20455 ts_tablet_manager.cc:1398] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 01:55:50.207850 16511 tablet_copy_service.cc:342] P 39059854b64045698ded1ba4e8d47304: Request end of tablet copy session aa7ec9abef1747e5a9e2404cee265c20-1fa6224f34ef4bc58797f6cf801ff6da received from {username='slave'} at 127.31.161.202:34163
I20240502 01:55:50.207983 16511 tablet_copy_service.cc:433] P 39059854b64045698ded1ba4e8d47304: ending tablet copy session aa7ec9abef1747e5a9e2404cee265c20-1fa6224f34ef4bc58797f6cf801ff6da on tablet 1fa6224f34ef4bc58797f6cf801ff6da with peer aa7ec9abef1747e5a9e2404cee265c20
I20240502 01:55:50.210743 18913 raft_consensus.cc:1203] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 01:55:50.211410 20347 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.212139 16752 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.212404 16882 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.216199 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a8f7e3c8b6d6434db2a99d7139599a5d with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:50.217415 19782 raft_consensus.cc:1050] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304: attempting to promote NON_VOTER a7c4573db86a415285d274ca57dde659 to VOTER
I20240502 01:55:50.218062 16082 catalog_manager.cc:5484] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a reported cstate change: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.218454 19782 consensus_queue.cc:237] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:50.221417 16358 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.221949 20526 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.222079 16225 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.222637 20526 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.223461 16358 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.224215 16471 tablet_service.cc:1514] Processing DeleteTablet for tablet a8f7e3c8b6d6434db2a99d7139599a5d with delete_type TABLET_DATA_TOMBSTONED (TS 39059854b64045698ded1ba4e8d47304 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51446
I20240502 01:55:50.225231 20567 tablet_replica.cc:330] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304: stopping tablet replica
I20240502 01:55:50.226023 16225 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.227658 18647 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.225719 20567 raft_consensus.cc:2229] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:50.229015 20567 raft_consensus.cc:2258] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:50.233939 18913 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.244166 20567 ts_tablet_manager.cc:1875] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:50.249974 17013 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.250793 18647 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.251120 16621 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.251328 20323 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.251889 18780 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.251940 20323 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.253968 20315 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.254555 20315 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.258371 20567 ts_tablet_manager.cc:1888] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 01:55:50.263180 20567 log.cc:1196] T a8f7e3c8b6d6434db2a99d7139599a5d P 39059854b64045698ded1ba4e8d47304: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/wals/a8f7e3c8b6d6434db2a99d7139599a5d
I20240502 01:55:50.264216 16067 catalog_manager.cc:4830] TS 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693): tablet a8f7e3c8b6d6434db2a99d7139599a5d (table rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]) successfully deleted
I20240502 01:55:50.286181 20399 tablet_copy_client.cc:308] T cec84c633b0a4c4fb6e074c79432fe61 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:40425
I20240502 01:55:50.286440 20323 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.290031 16642 tablet_copy_service.cc:140] P 703f6f043c6041a4b629933690580367: Received BeginTabletCopySession request for tablet cec84c633b0a4c4fb6e074c79432fe61 from peer 0982e15b4c74425cad8d7d642eb63316 ({username='slave'} at 127.31.161.200:36469)
I20240502 01:55:50.290208 16642 tablet_copy_service.cc:161] P 703f6f043c6041a4b629933690580367: Beginning new tablet copy session on tablet cec84c633b0a4c4fb6e074c79432fe61 from peer 0982e15b4c74425cad8d7d642eb63316 at {username='slave'} at 127.31.161.200:36469: session id = 0982e15b4c74425cad8d7d642eb63316-cec84c633b0a4c4fb6e074c79432fe61
I20240502 01:55:50.290290 17015 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.293051 16642 tablet_copy_source_session.cc:205] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:50.296280 20399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec84c633b0a4c4fb6e074c79432fe61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:50.296475 16884 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.297103 20315 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.300412 16225 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.301179 16082 catalog_manager.cc:5484] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.301656 20315 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.302498 18645 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.311169 18780 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.311702 16225 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.313810 20399 tablet_copy_client.cc:783] T cec84c633b0a4c4fb6e074c79432fe61 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:50.314010 20399 tablet_copy_client.cc:647] T cec84c633b0a4c4fb6e074c79432fe61 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:50.315064 16358 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.316632 16621 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.318509 20402 ts_tablet_manager.cc:902] T b55b98b06aca47988b14f486973cc370 P a7c4573db86a415285d274ca57dde659: Initiating tablet copy from peer ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825)
I20240502 01:55:50.321285 20399 tablet_copy_client.cc:523] T cec84c633b0a4c4fb6e074c79432fe61 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:50.324365 19782 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.326243 20399 tablet_bootstrap.cc:492] T cec84c633b0a4c4fb6e074c79432fe61 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap starting.
I20240502 01:55:50.328214 16884 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.337118 20578 raft_consensus.cc:1050] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304: attempting to promote NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 to VOTER
I20240502 01:55:50.338953 20578 consensus_queue.cc:237] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:50.344020 18913 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.344880 20323 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.348208 16358 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.356492 16225 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.357436 17013 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.358320 20323 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.358417 18647 raft_consensus.cc:1203] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 01:55:50.358664 20570 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.360440 16884 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.361119 20323 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.364190 18780 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.364385 16752 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.364818 20578 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.365248 20323 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.369719 16358 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.371104 20323 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.384966 18647 raft_consensus.cc:1203] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 01:55:50.386775 20323 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.391758 18912 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.395344 16751 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.398167 16075 catalog_manager.cc:5484] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316 reported cstate change: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.405525 16884 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.405534 16225 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.411795 17013 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.417693 16488 consensus_queue.cc:237] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:50.417356 16358 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.424954 18913 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.425535 16752 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.425576 16882 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.425563 18780 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.426311 20323 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.429466 20578 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.429962 20578 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.430369 20578 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.433310 18780 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.434943 16223 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.436079 20323 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.439837 20570 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.441416 16223 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.447577 16076 catalog_manager.cc:5484] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e reported cstate change: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.447572 18779 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.450788 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 18e676bcd8b04f18a6814d6ebbd1b488 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:50.452322 18913 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.456837 16339 tablet_service.cc:1514] Processing DeleteTablet for tablet 18e676bcd8b04f18a6814d6ebbd1b488 with delete_type TABLET_DATA_TOMBSTONED (TS dc9fcb8d12cb4ab28ec821c9473adeeb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43508
I20240502 01:55:50.458675 17015 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.459434 20578 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.461936 16884 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.464355 17013 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.473958 16752 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.488334 20598 ts_tablet_manager.cc:902] T bc9cc69a319d46b8940f4c7dc2e7e656 P 0982e15b4c74425cad8d7d642eb63316: Initiating tablet copy from peer 703f6f043c6041a4b629933690580367 (127.31.161.196:40425)
I20240502 01:55:50.490746 20402 tablet_copy_client.cc:308] T b55b98b06aca47988b14f486973cc370 P a7c4573db86a415285d274ca57dde659: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.197:46825
I20240502 01:55:50.491554 16774 tablet_copy_service.cc:140] P ed2e7e4188d0400b8689e28bbf5096ad: Received BeginTabletCopySession request for tablet b55b98b06aca47988b14f486973cc370 from peer a7c4573db86a415285d274ca57dde659 ({username='slave'} at 127.31.161.201:44943)
I20240502 01:55:50.491752 16774 tablet_copy_service.cc:161] P ed2e7e4188d0400b8689e28bbf5096ad: Beginning new tablet copy session on tablet b55b98b06aca47988b14f486973cc370 from peer a7c4573db86a415285d274ca57dde659 at {username='slave'} at 127.31.161.201:44943: session id = a7c4573db86a415285d274ca57dde659-b55b98b06aca47988b14f486973cc370
I20240502 01:55:50.494832 16774 tablet_copy_source_session.cc:205] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:50.496692 20402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55b98b06aca47988b14f486973cc370. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:50.503270 20597 tablet_replica.cc:330] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb: stopping tablet replica
I20240502 01:55:50.503712 20402 tablet_copy_client.cc:783] T b55b98b06aca47988b14f486973cc370 P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:50.503929 20402 tablet_copy_client.cc:647] T b55b98b06aca47988b14f486973cc370 P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:50.505647 20402 tablet_copy_client.cc:523] T b55b98b06aca47988b14f486973cc370 P a7c4573db86a415285d274ca57dde659: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:50.506284 20597 raft_consensus.cc:2229] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:50.507112 20597 raft_consensus.cc:2258] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:50.511160 20402 tablet_bootstrap.cc:492] T b55b98b06aca47988b14f486973cc370 P a7c4573db86a415285d274ca57dde659: Bootstrap starting.
I20240502 01:55:50.512523 20597 ts_tablet_manager.cc:1875] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:50.520263 20391 raft_consensus.cc:1050] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367: attempting to promote NON_VOTER 0982e15b4c74425cad8d7d642eb63316 to VOTER
I20240502 01:55:50.521607 20391 consensus_queue.cc:237] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:50.526685 20399 tablet_bootstrap.cc:492] T cec84c633b0a4c4fb6e074c79432fe61 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:50.527192 20399 tablet_bootstrap.cc:492] T cec84c633b0a4c4fb6e074c79432fe61 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap complete.
I20240502 01:55:50.528822 18647 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.529534 16488 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.529768 20387 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.531108 16358 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.531522 16223 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.531877 19910 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.532428 19910 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.534720 20603 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.536296 16884 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.537488 17013 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.540760 20387 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.541402 20387 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.555269 20597 ts_tablet_manager.cc:1888] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 01:55:50.555537 20597 log.cc:1196] T 18e676bcd8b04f18a6814d6ebbd1b488 P dc9fcb8d12cb4ab28ec821c9473adeeb: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/wals/18e676bcd8b04f18a6814d6ebbd1b488
I20240502 01:55:50.556589 16070 catalog_manager.cc:4830] TS dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133): tablet 18e676bcd8b04f18a6814d6ebbd1b488 (table rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]) successfully deleted
I20240502 01:55:50.566536 20496 raft_consensus.cc:1050] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad: attempting to promote NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 to VOTER
I20240502 01:55:50.566928 18779 raft_consensus.cc:1203] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 01:55:50.567766 20496 consensus_queue.cc:237] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:50.573786 18779 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.574599 17015 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.575137 19881 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.575688 19881 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.580114 16621 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.580690 20476 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.580888 16489 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.583191 20476 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.587694 16358 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.588399 16223 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.589149 20476 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.603374 20402 tablet_bootstrap.cc:492] T b55b98b06aca47988b14f486973cc370 P a7c4573db86a415285d274ca57dde659: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:50.603375 20496 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.604002 20402 tablet_bootstrap.cc:492] T b55b98b06aca47988b14f486973cc370 P a7c4573db86a415285d274ca57dde659: Bootstrap complete.
I20240502 01:55:50.604365 20402 ts_tablet_manager.cc:1367] T b55b98b06aca47988b14f486973cc370 P a7c4573db86a415285d274ca57dde659: Time spent bootstrapping tablet: real 0.093s user 0.046s sys 0.012s
I20240502 01:55:50.605257 20402 raft_consensus.cc:348] T b55b98b06aca47988b14f486973cc370 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:50.605680 20402 raft_consensus.cc:729] T b55b98b06aca47988b14f486973cc370 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a7c4573db86a415285d274ca57dde659, State: Initialized, Role: LEARNER
I20240502 01:55:50.605966 20402 consensus_queue.cc:260] T b55b98b06aca47988b14f486973cc370 P a7c4573db86a415285d274ca57dde659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:50.606716 20402 ts_tablet_manager.cc:1398] T b55b98b06aca47988b14f486973cc370 P a7c4573db86a415285d274ca57dde659: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:50.607554 16774 tablet_copy_service.cc:342] P ed2e7e4188d0400b8689e28bbf5096ad: Request end of tablet copy session a7c4573db86a415285d274ca57dde659-b55b98b06aca47988b14f486973cc370 received from {username='slave'} at 127.31.161.201:44943
I20240502 01:55:50.607686 16774 tablet_copy_service.cc:433] P ed2e7e4188d0400b8689e28bbf5096ad: ending tablet copy session a7c4573db86a415285d274ca57dde659-b55b98b06aca47988b14f486973cc370 on tablet b55b98b06aca47988b14f486973cc370 with peer a7c4573db86a415285d274ca57dde659
I20240502 01:55:50.527494 20399 ts_tablet_manager.cc:1367] T cec84c633b0a4c4fb6e074c79432fe61 P 0982e15b4c74425cad8d7d642eb63316: Time spent bootstrapping tablet: real 0.201s user 0.048s sys 0.006s
I20240502 01:55:50.528919 18913 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.534755 20598 tablet_copy_client.cc:308] T bc9cc69a319d46b8940f4c7dc2e7e656 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:40425
I20240502 01:55:50.574160 18912 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.610330 20399 raft_consensus.cc:348] T cec84c633b0a4c4fb6e074c79432fe61 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:50.610636 20399 raft_consensus.cc:729] T cec84c633b0a4c4fb6e074c79432fe61 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0982e15b4c74425cad8d7d642eb63316, State: Initialized, Role: LEARNER
I20240502 01:55:50.610266 20603 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.610913 20399 consensus_queue.cc:260] T cec84c633b0a4c4fb6e074c79432fe61 P 0982e15b4c74425cad8d7d642eb63316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:50.611563 20399 ts_tablet_manager.cc:1398] T cec84c633b0a4c4fb6e074c79432fe61 P 0982e15b4c74425cad8d7d642eb63316: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:55:50.615126 20620 ts_tablet_manager.cc:902] T 297561b66dd7472ab817c2a23723565c P a7c4573db86a415285d274ca57dde659: Initiating tablet copy from peer a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:50.617800 20612 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.618602 16488 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.621165 16357 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.621851 19881 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 01:55:50.623692 16557 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:41953, user_credentials={real_user=slave}} blocked reactor thread for 91539.3us
I20240502 01:55:50.628157 16082 catalog_manager.cc:5484] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 reported cstate change: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:50.628310 16884 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.633535 19881 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.637465 17013 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.638468 16358 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.642884 16489 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.649246 16082 catalog_manager.cc:5484] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.653771 16642 tablet_copy_service.cc:140] P 703f6f043c6041a4b629933690580367: Received BeginTabletCopySession request for tablet bc9cc69a319d46b8940f4c7dc2e7e656 from peer 0982e15b4c74425cad8d7d642eb63316 ({username='slave'} at 127.31.161.200:36469)
I20240502 01:55:50.653935 16642 tablet_copy_service.cc:161] P 703f6f043c6041a4b629933690580367: Beginning new tablet copy session on tablet bc9cc69a319d46b8940f4c7dc2e7e656 from peer 0982e15b4c74425cad8d7d642eb63316 at {username='slave'} at 127.31.161.200:36469: session id = 0982e15b4c74425cad8d7d642eb63316-bc9cc69a319d46b8940f4c7dc2e7e656
I20240502 01:55:50.657248 16641 tablet_copy_service.cc:342] P 703f6f043c6041a4b629933690580367: Request end of tablet copy session 0982e15b4c74425cad8d7d642eb63316-cec84c633b0a4c4fb6e074c79432fe61 received from {username='slave'} at 127.31.161.200:36469
I20240502 01:55:50.657392 16641 tablet_copy_service.cc:433] P 703f6f043c6041a4b629933690580367: ending tablet copy session 0982e15b4c74425cad8d7d642eb63316-cec84c633b0a4c4fb6e074c79432fe61 on tablet cec84c633b0a4c4fb6e074c79432fe61 with peer 0982e15b4c74425cad8d7d642eb63316
I20240502 01:55:50.666107 16642 tablet_copy_source_session.cc:205] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:50.668574 16621 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.672283 20578 raft_consensus.cc:1050] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304: attempting to promote NON_VOTER 0982e15b4c74425cad8d7d642eb63316 to VOTER
I20240502 01:55:50.673501 20578 consensus_queue.cc:237] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:50.676822 17013 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.691686 20598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc9cc69a319d46b8940f4c7dc2e7e656. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 01:55:50.700573 16688 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.200:38559, user_credentials={real_user=slave}} blocked reactor thread for 62628.5us
I20240502 01:55:50.701508 18779 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.707626 20598 tablet_copy_client.cc:783] T bc9cc69a319d46b8940f4c7dc2e7e656 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:50.707983 20598 tablet_copy_client.cc:647] T bc9cc69a319d46b8940f4c7dc2e7e656 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:50.709093 18913 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.713841 18645 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.722000 20598 tablet_copy_client.cc:523] T bc9cc69a319d46b8940f4c7dc2e7e656 P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:50.728273 20598 tablet_bootstrap.cc:492] T bc9cc69a319d46b8940f4c7dc2e7e656 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap starting.
I20240502 01:55:50.736847 16225 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.738379 18645 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.739172 20570 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.741113 16223 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.741565 16358 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.742189 20570 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.744607 20323 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.730543 18913 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.745645 20570 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.745338 16222 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.748500 16621 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.749045 20570 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.753770 16752 consensus_queue.cc:237] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:50.756922 16620 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.757699 16488 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.757961 20496 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.758345 20637 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.762375 20476 raft_consensus.cc:1050] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad: attempting to promote NON_VOTER a7c4573db86a415285d274ca57dde659 to VOTER
I20240502 01:55:50.764003 20420 raft_consensus.cc:1050] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a: attempting to promote NON_VOTER a7c4573db86a415285d274ca57dde659 to VOTER
I20240502 01:55:50.765177 20420 consensus_queue.cc:237] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:50.765817 17015 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.767563 20578 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.768455 16751 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.763535 20476 consensus_queue.cc:237] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:50.770311 18645 raft_consensus.cc:1203] T b55b98b06aca47988b14f486973cc370 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 01:55:50.770572 20570 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.775414 18911 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.776089 20596 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.776372 16357 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.776499 17016 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.777228 16224 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.777237 20596 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.778232 16620 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.778194 20585 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.778730 20585 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.780392 20585 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240502 01:55:50.782801 16488 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.783367 20585 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.784142 18645 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.784693 20585 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.798440 18910 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.798682 18779 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.799268 19881 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.799782 19881 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.807632 20637 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.811843 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dd96134671e1487eba32aaa1dcd6cd67 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:50.813828 18779 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.815661 18909 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.817447 17013 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.818128 19881 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.822772 16223 raft_consensus.cc:1261] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.826293 18912 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.829488 20650 consensus_queue.cc:1035] T dd96134671e1487eba32aaa1dcd6cd67 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.822400 16076 catalog_manager.cc:5484] T dd96134671e1487eba32aaa1dcd6cd67 P aa7ec9abef1747e5a9e2404cee265c20 reported cstate change: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.832158 17013 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.834240 16488 consensus_queue.cc:237] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:50.836154 16225 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.837293 18645 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.838483 16884 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.838969 16339 tablet_service.cc:1514] Processing DeleteTablet for tablet dd96134671e1487eba32aaa1dcd6cd67 with delete_type TABLET_DATA_TOMBSTONED (TS dc9fcb8d12cb4ab28ec821c9473adeeb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43508
I20240502 01:55:50.839078 20086 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.839608 20597 tablet_replica.cc:330] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb: stopping tablet replica
I20240502 01:55:50.839821 20597 raft_consensus.cc:2229] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:50.839823 20578 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.840168 20597 raft_consensus.cc:2258] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:50.840909 16619 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.842989 17015 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.843008 20597 ts_tablet_manager.cc:1875] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:50.843586 20323 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.844007 20323 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.845194 16619 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.850167 20597 ts_tablet_manager.cc:1888] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 01:55:50.850360 20597 log.cc:1196] T dd96134671e1487eba32aaa1dcd6cd67 P dc9fcb8d12cb4ab28ec821c9473adeeb: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/wals/dd96134671e1487eba32aaa1dcd6cd67
I20240502 01:55:50.852334 18645 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.852543 16489 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.852924 16070 catalog_manager.cc:4830] TS dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133): tablet dd96134671e1487eba32aaa1dcd6cd67 (table rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]) successfully deleted
I20240502 01:55:50.853245 20496 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.852882 16488 raft_consensus.cc:2932] T dd96134671e1487eba32aaa1dcd6cd67 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:50.855028 16359 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.856571 20496 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.856653 17014 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.857156 20496 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.858011 16357 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.858424 18909 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.858762 16223 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.859071 20496 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.860149 20323 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.860292 19881 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.863732 20323 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.872217 17015 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.875958 16357 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.876781 18779 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.878280 19782 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.879055 19782 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.880738 20585 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.884214 20620 tablet_copy_client.cc:308] T 297561b66dd7472ab817c2a23723565c P a7c4573db86a415285d274ca57dde659: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.199:46837
I20240502 01:55:50.883832 17016 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.884897 17036 tablet_copy_service.cc:140] P a6575ebf978a4631a13ba5c243c3c4f9: Received BeginTabletCopySession request for tablet 297561b66dd7472ab817c2a23723565c from peer a7c4573db86a415285d274ca57dde659 ({username='slave'} at 127.31.161.201:32955)
I20240502 01:55:50.885221 17036 tablet_copy_service.cc:161] P a6575ebf978a4631a13ba5c243c3c4f9: Beginning new tablet copy session on tablet 297561b66dd7472ab817c2a23723565c from peer a7c4573db86a415285d274ca57dde659 at {username='slave'} at 127.31.161.201:32955: session id = a7c4573db86a415285d274ca57dde659-297561b66dd7472ab817c2a23723565c
I20240502 01:55:50.885355 18646 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.885607 16222 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.888206 17036 tablet_copy_source_session.cc:205] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:50.888844 17012 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.890293 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:50.891750 16884 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.892007 18646 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.892652 16356 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.900097 16224 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.901894 16081 catalog_manager.cc:5484] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:50.897728 16075 catalog_manager.cc:5484] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a reported cstate change: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6b9f9aaa1c86420190af75270914449a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:50.908246 18910 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.911767 16621 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.916635 16882 consensus_queue.cc:237] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:50.916793 18647 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.917454 16619 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.921248 16620 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.922675 16355 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.923688 18646 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.923893 20596 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.924484 20596 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.924877 16488 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.927203 16356 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:50.929369 16884 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:50.930995 18909 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.931577 20596 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.931473 18779 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.937721 20637 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.937551 20585 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.938657 20637 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:50.940531 18909 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.941287 16075 catalog_manager.cc:5484] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a reported cstate change: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.941823 16068 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 453dff7b204a431592ac15c058db0493 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:50.947269 16620 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.947795 20596 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.948535 16083 catalog_manager.cc:5484] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316 reported cstate change: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "6b9f9aaa1c86420190af75270914449a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.948344 16356 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.948475 18647 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.949441 16620 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.963835 16204 tablet_service.cc:1514] Processing DeleteTablet for tablet 453dff7b204a431592ac15c058db0493 with delete_type TABLET_DATA_TOMBSTONED (TS cdd4d50250184223992325c86cb6c68e not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:40596
I20240502 01:55:50.964831 20676 tablet_replica.cc:330] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e: stopping tablet replica
I20240502 01:55:50.968024 20676 raft_consensus.cc:2229] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:50.969455 20676 raft_consensus.cc:2258] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:50.970453 16204 tablet_service.cc:1514] Processing DeleteTablet for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c with delete_type TABLET_DATA_TOMBSTONED (TS cdd4d50250184223992325c86cb6c68e not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:40596
I20240502 01:55:50.971993 17014 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:50.972226 20676 ts_tablet_manager.cc:1875] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:50.972833 20654 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:50.974706 17014 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:50.981237 20676 ts_tablet_manager.cc:1888] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 01:55:50.981447 20676 log.cc:1196] T 453dff7b204a431592ac15c058db0493 P cdd4d50250184223992325c86cb6c68e: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/453dff7b204a431592ac15c058db0493
I20240502 01:55:50.982599 16069 catalog_manager.cc:4830] TS cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953): tablet 453dff7b204a431592ac15c058db0493 (table rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]) successfully deleted
I20240502 01:55:50.982947 20676 tablet_replica.cc:330] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e: stopping tablet replica
I20240502 01:55:50.983217 20676 raft_consensus.cc:2229] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:50.983657 20676 raft_consensus.cc:2258] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:50.985976 20620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 297561b66dd7472ab817c2a23723565c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:50.991598 18912 raft_consensus.cc:1203] T cec84c633b0a4c4fb6e074c79432fe61 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 01:55:51.000938 20676 ts_tablet_manager.cc:1875] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:51.001000 18913 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.001822 20650 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.008239 16356 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.009348 16223 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.011902 20676 ts_tablet_manager.cc:1888] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 01:55:51.012715 17014 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.014683 20676 log.cc:1196] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P cdd4d50250184223992325c86cb6c68e: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/6150a46aa8ca4d9bbd16c141ad2f1a3c
I20240502 01:55:51.015413 16069 catalog_manager.cc:4830] TS cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953): tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c (table rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]) successfully deleted
I20240502 01:55:51.016861 16882 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.025369 16489 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.027933 16079 catalog_manager.cc:5484] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a reported cstate change: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.030230 20620 tablet_copy_client.cc:783] T 297561b66dd7472ab817c2a23723565c P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:51.030442 20620 tablet_copy_client.cc:647] T 297561b66dd7472ab817c2a23723565c P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:51.038097 16751 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.039273 16486 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.039992 20654 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.040187 18911 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.041841 16486 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.046986 18645 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.053642 20620 tablet_copy_client.cc:523] T 297561b66dd7472ab817c2a23723565c P a7c4573db86a415285d274ca57dde659: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:51.057704 16751 consensus_queue.cc:237] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:51.062619 20620 tablet_bootstrap.cc:492] T 297561b66dd7472ab817c2a23723565c P a7c4573db86a415285d274ca57dde659: Bootstrap starting.
I20240502 01:55:51.066151 20323 raft_consensus.cc:1050] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304: attempting to promote NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 to VOTER
I20240502 01:55:51.069165 18647 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.069736 20650 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.070335 20323 consensus_queue.cc:237] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:51.072538 18911 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.073256 16358 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.073804 17014 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.073868 20650 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.074822 20637 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.078895 16882 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.079471 20637 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.080951 20650 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.081514 16356 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.082430 16882 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.085273 16488 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.085994 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 512eab02225d4544a342811f0a7359aa with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:51.085577 18645 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.086776 17014 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.080444 20598 tablet_bootstrap.cc:492] T bc9cc69a319d46b8940f4c7dc2e7e656 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:51.090200 20598 tablet_bootstrap.cc:492] T bc9cc69a319d46b8940f4c7dc2e7e656 P 0982e15b4c74425cad8d7d642eb63316: Bootstrap complete.
I20240502 01:55:51.090489 20598 ts_tablet_manager.cc:1367] T bc9cc69a319d46b8940f4c7dc2e7e656 P 0982e15b4c74425cad8d7d642eb63316: Time spent bootstrapping tablet: real 0.362s user 0.047s sys 0.003s
I20240502 01:55:51.091244 20598 raft_consensus.cc:348] T bc9cc69a319d46b8940f4c7dc2e7e656 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:51.091583 20598 raft_consensus.cc:729] T bc9cc69a319d46b8940f4c7dc2e7e656 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0982e15b4c74425cad8d7d642eb63316, State: Initialized, Role: LEARNER
I20240502 01:55:51.091802 20598 consensus_queue.cc:260] T bc9cc69a319d46b8940f4c7dc2e7e656 P 0982e15b4c74425cad8d7d642eb63316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:51.092494 20598 ts_tablet_manager.cc:1398] T bc9cc69a319d46b8940f4c7dc2e7e656 P 0982e15b4c74425cad8d7d642eb63316: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 01:55:51.093171 20638 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.093668 20638 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.096122 16488 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.096763 16076 catalog_manager.cc:5484] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.101661 16642 tablet_copy_service.cc:342] P 703f6f043c6041a4b629933690580367: Request end of tablet copy session 0982e15b4c74425cad8d7d642eb63316-bc9cc69a319d46b8940f4c7dc2e7e656 received from {username='slave'} at 127.31.161.200:36469
I20240502 01:55:51.101806 16642 tablet_copy_service.cc:433] P 703f6f043c6041a4b629933690580367: ending tablet copy session 0982e15b4c74425cad8d7d642eb63316-bc9cc69a319d46b8940f4c7dc2e7e656 on tablet bc9cc69a319d46b8940f4c7dc2e7e656 with peer 0982e15b4c74425cad8d7d642eb63316
I20240502 01:55:51.103796 18779 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.103425 18911 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.110288 20578 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.111204 16204 tablet_service.cc:1514] Processing DeleteTablet for tablet 512eab02225d4544a342811f0a7359aa with delete_type TABLET_DATA_TOMBSTONED (TS cdd4d50250184223992325c86cb6c68e not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:40596
I20240502 01:55:51.111670 20676 tablet_replica.cc:330] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e: stopping tablet replica
I20240502 01:55:51.111868 20676 raft_consensus.cc:2229] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:51.112121 20676 raft_consensus.cc:2258] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:51.113437 20676 ts_tablet_manager.cc:1875] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:51.113624 17014 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.114286 19782 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.117560 18911 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.118134 19782 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.122247 20676 ts_tablet_manager.cc:1888] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 01:55:51.124099 20676 log.cc:1196] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/512eab02225d4544a342811f0a7359aa
I20240502 01:55:51.125005 16069 catalog_manager.cc:4830] TS cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953): tablet 512eab02225d4544a342811f0a7359aa (table rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]) successfully deleted
I20240502 01:55:51.129611 16882 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.130137 19782 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.134848 16620 consensus_queue.cc:237] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:51.138753 16752 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.139370 19782 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.140224 16356 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.141201 18645 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.142154 16223 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.142969 18911 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.143232 20634 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.144416 20391 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.144904 20391 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.145385 20391 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.151845 20634 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.152407 16224 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.153281 20578 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.155117 16356 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.160164 19782 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.162919 16223 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.165383 16356 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.171064 17016 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.171645 20614 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.179534 16083 catalog_manager.cc:5484] T 6cde24f949334006aa9e241e844e81eb P cdd4d50250184223992325c86cb6c68e reported cstate change: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.182904 16070 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6cde24f949334006aa9e241e844e81eb with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:51.184186 16471 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cde24f949334006aa9e241e844e81eb with delete_type TABLET_DATA_TOMBSTONED (TS 39059854b64045698ded1ba4e8d47304 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51446
I20240502 01:55:51.192706 17014 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.203075 20578 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:51.210485 17016 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:51.210515 18913 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:51.213335 16883 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:51.220607 18911 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.221578 16083 catalog_manager.cc:5484] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316 reported cstate change: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:51.221451 18645 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.222606 16883 raft_consensus.cc:1261] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.223289 20612 consensus_queue.cc:1035] T 6cde24f949334006aa9e241e844e81eb P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.226963 18779 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:51.231935 16882 raft_consensus.cc:2932] T 6cde24f949334006aa9e241e844e81eb P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.238526 20693 tablet_replica.cc:330] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304: stopping tablet replica
I20240502 01:55:51.239445 20693 raft_consensus.cc:2229] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:51.241020 16358 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:51.242106 20693 raft_consensus.cc:2258] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:51.243300 20693 ts_tablet_manager.cc:1875] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:51.254113 16223 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:51.256214 20620 tablet_bootstrap.cc:492] T 297561b66dd7472ab817c2a23723565c P a7c4573db86a415285d274ca57dde659: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:51.256683 20620 tablet_bootstrap.cc:492] T 297561b66dd7472ab817c2a23723565c P a7c4573db86a415285d274ca57dde659: Bootstrap complete.
I20240502 01:55:51.256987 20620 ts_tablet_manager.cc:1367] T 297561b66dd7472ab817c2a23723565c P a7c4573db86a415285d274ca57dde659: Time spent bootstrapping tablet: real 0.194s user 0.054s sys 0.003s
I20240502 01:55:51.259717 20693 ts_tablet_manager.cc:1888] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 01:55:51.262010 20620 raft_consensus.cc:348] T 297561b66dd7472ab817c2a23723565c P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:51.262400 20620 raft_consensus.cc:729] T 297561b66dd7472ab817c2a23723565c P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a7c4573db86a415285d274ca57dde659, State: Initialized, Role: LEARNER
I20240502 01:55:51.262589 20693 log.cc:1196] T 6cde24f949334006aa9e241e844e81eb P 39059854b64045698ded1ba4e8d47304: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/wals/6cde24f949334006aa9e241e844e81eb
I20240502 01:55:51.262642 20620 consensus_queue.cc:260] T 297561b66dd7472ab817c2a23723565c P a7c4573db86a415285d274ca57dde659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:51.263929 16751 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:51.263754 16067 catalog_manager.cc:4830] TS 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693): tablet 6cde24f949334006aa9e241e844e81eb (table rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]) successfully deleted
I20240502 01:55:51.270359 20620 ts_tablet_manager.cc:1398] T 297561b66dd7472ab817c2a23723565c P a7c4573db86a415285d274ca57dde659: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 01:55:51.271023 17036 tablet_copy_service.cc:342] P a6575ebf978a4631a13ba5c243c3c4f9: Request end of tablet copy session a7c4573db86a415285d274ca57dde659-297561b66dd7472ab817c2a23723565c received from {username='slave'} at 127.31.161.201:32955
I20240502 01:55:51.271162 17036 tablet_copy_service.cc:433] P a6575ebf978a4631a13ba5c243c3c4f9: ending tablet copy session a7c4573db86a415285d274ca57dde659-297561b66dd7472ab817c2a23723565c on tablet 297561b66dd7472ab817c2a23723565c with peer a7c4573db86a415285d274ca57dde659
I20240502 01:55:51.275211 16488 consensus_queue.cc:237] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:51.278439 18779 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.278456 18913 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.279026 20570 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.279708 20323 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.284274 17014 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.284822 16884 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.285142 20315 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.285470 16224 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.285524 16752 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.286110 20315 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.287333 20315 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.287994 20315 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:51.288619 17014 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:51.289757 16884 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:51.292579 19782 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.293608 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1fa6224f34ef4bc58797f6cf801ff6da with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:51.294731 18913 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:51.294903 16083 catalog_manager.cc:5484] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:51.295591 16224 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:51.296346 16754 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:51.300096 16339 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fa6224f34ef4bc58797f6cf801ff6da with delete_type TABLET_DATA_TOMBSTONED (TS dc9fcb8d12cb4ab28ec821c9473adeeb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43508
I20240502 01:55:51.300474 20597 tablet_replica.cc:330] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb: stopping tablet replica
I20240502 01:55:51.300693 20597 raft_consensus.cc:2229] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:51.300976 20597 raft_consensus.cc:2258] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:51.303071 20597 ts_tablet_manager.cc:1875] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:51.306102 18779 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:51.309394 20637 raft_consensus.cc:1050] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad: attempting to promote NON_VOTER a7c4573db86a415285d274ca57dde659 to VOTER
I20240502 01:55:51.310503 20637 consensus_queue.cc:237] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:51.310815 20597 ts_tablet_manager.cc:1888] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 01:55:51.311084 20597 log.cc:1196] T 1fa6224f34ef4bc58797f6cf801ff6da P dc9fcb8d12cb4ab28ec821c9473adeeb: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/wals/1fa6224f34ef4bc58797f6cf801ff6da
I20240502 01:55:51.311939 16070 catalog_manager.cc:4830] TS dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133): tablet 1fa6224f34ef4bc58797f6cf801ff6da (table rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]) successfully deleted
I20240502 01:55:51.317797 16620 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.318915 16488 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.319561 16358 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.319707 20637 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.319828 18645 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.320443 20637 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.320660 17014 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.320760 20638 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.321225 20650 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.322137 16224 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.323452 20496 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.323935 18913 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.323948 20496 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.325671 20637 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.328465 18911 raft_consensus.cc:1203] T bc9cc69a319d46b8940f4c7dc2e7e656 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 01:55:51.339320 20637 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.344986 16620 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.342208 18913 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.351646 18647 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.353315 17016 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.355060 16224 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.356714 16076 catalog_manager.cc:5484] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316 reported cstate change: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.357941 16488 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.367529 16358 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.375061 16754 consensus_queue.cc:237] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:51.378593 16620 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.378664 18913 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.378752 16358 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.379319 16488 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.379395 20476 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.379792 20650 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.380180 17016 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.380311 20650 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.380810 20650 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.380911 18647 raft_consensus.cc:1261] T b55b98b06aca47988b14f486973cc370 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:51.381479 20637 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.381953 20637 consensus_queue.cc:1035] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20240502 01:55:51.384604 19881 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.385442 18913 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.385932 18647 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.388396 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b55b98b06aca47988b14f486973cc370 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:51.388418 16620 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.388883 16488 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.389513 17014 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.395699 16076 catalog_manager.cc:5484] T b55b98b06aca47988b14f486973cc370 P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.395866 16358 raft_consensus.cc:2932] T b55b98b06aca47988b14f486973cc370 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:51.401901 16204 tablet_service.cc:1514] Processing DeleteTablet for tablet b55b98b06aca47988b14f486973cc370 with delete_type TABLET_DATA_TOMBSTONED (TS cdd4d50250184223992325c86cb6c68e not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:40596
I20240502 01:55:51.402416 20676 tablet_replica.cc:330] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e: stopping tablet replica
I20240502 01:55:51.402652 20676 raft_consensus.cc:2229] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:51.402920 20676 raft_consensus.cc:2258] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:51.404001 20676 ts_tablet_manager.cc:1875] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:51.412009 20676 ts_tablet_manager.cc:1888] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 01:55:51.412243 20676 log.cc:1196] T b55b98b06aca47988b14f486973cc370 P cdd4d50250184223992325c86cb6c68e: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/b55b98b06aca47988b14f486973cc370
I20240502 01:55:51.413173 16069 catalog_manager.cc:4830] TS cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953): tablet b55b98b06aca47988b14f486973cc370 (table rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]) successfully deleted
I20240502 01:55:51.420481 16488 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "48a81a0b7d47479e929d80041f34a3b1"
dest_uuid: "39059854b64045698ded1ba4e8d47304"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52690
I20240502 01:55:51.420742 16488 raft_consensus.cc:595] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Received request to transfer leadership
I20240502 01:55:51.439165 20323 raft_consensus.cc:979] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304: : Instructing follower a7c4573db86a415285d274ca57dde659 to start an election
I20240502 01:55:51.439317 20323 raft_consensus.cc:1067] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Signalling peer a7c4573db86a415285d274ca57dde659 to start an election
I20240502 01:55:51.440245 18647 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "48a81a0b7d47479e929d80041f34a3b1"
dest_uuid: "a7c4573db86a415285d274ca57dde659"
from {username='slave'} at 127.31.161.195:39547
I20240502 01:55:51.440615 18647 raft_consensus.cc:482] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 01:55:51.440819 18647 raft_consensus.cc:3037] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:51.443903 18647 raft_consensus.cc:504] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:51.448961 16754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a81a0b7d47479e929d80041f34a3b1" candidate_uuid: "a7c4573db86a415285d274ca57dde659" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "ed2e7e4188d0400b8689e28bbf5096ad"
I20240502 01:55:51.449229 16754 raft_consensus.cc:3037] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:51.452265 16754 raft_consensus.cc:2454] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7c4573db86a415285d274ca57dde659 in term 2.
I20240502 01:55:51.455101 17014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a81a0b7d47479e929d80041f34a3b1" candidate_uuid: "a7c4573db86a415285d274ca57dde659" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "a6575ebf978a4631a13ba5c243c3c4f9"
I20240502 01:55:51.455343 17014 raft_consensus.cc:3037] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:51.455541 16488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a81a0b7d47479e929d80041f34a3b1" candidate_uuid: "a7c4573db86a415285d274ca57dde659" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "39059854b64045698ded1ba4e8d47304"
I20240502 01:55:51.455502 16620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a81a0b7d47479e929d80041f34a3b1" candidate_uuid: "a7c4573db86a415285d274ca57dde659" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "703f6f043c6041a4b629933690580367"
I20240502 01:55:51.455740 16488 raft_consensus.cc:3032] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 01:55:51.455751 16620 raft_consensus.cc:3037] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:51.455873 16488 raft_consensus.cc:729] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39059854b64045698ded1ba4e8d47304, State: Running, Role: LEADER
I20240502 01:55:51.456279 16488 consensus_queue.cc:260] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:51.457054 16488 raft_consensus.cc:3037] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:51.458379 17014 raft_consensus.cc:2454] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7c4573db86a415285d274ca57dde659 in term 2.
I20240502 01:55:51.458652 16620 raft_consensus.cc:2454] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7c4573db86a415285d274ca57dde659 in term 2.
I20240502 01:55:51.460377 16488 raft_consensus.cc:2454] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7c4573db86a415285d274ca57dde659 in term 2.
I20240502 01:55:51.469460 20689 raft_consensus.cc:1050] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367: attempting to promote NON_VOTER 0982e15b4c74425cad8d7d642eb63316 to VOTER
I20240502 01:55:51.470852 20689 consensus_queue.cc:237] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:51.478778 18647 leader_election.cc:288] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [CANDIDATE]: Term 2 election: Requested vote from peers ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825), a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837), 703f6f043c6041a4b629933690580367 (127.31.161.196:40425), cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953), dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133), 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693), 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200:38559)
I20240502 01:55:51.487823 18913 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.488772 16488 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.489926 20614 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.491901 17014 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.493491 20634 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.495250 20614 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.497341 18582 leader_election.cc:302] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 8 voters: 5 yes votes; 0 no votes. yes voters: 39059854b64045698ded1ba4e8d47304, 703f6f043c6041a4b629933690580367, a6575ebf978a4631a13ba5c243c3c4f9, a7c4573db86a415285d274ca57dde659, ed2e7e4188d0400b8689e28bbf5096ad; no voters:
I20240502 01:55:51.500792 20293 raft_consensus.cc:2781] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 01:55:51.501318 20293 raft_consensus.cc:686] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [term 2 LEADER]: Becoming Leader. State: Replica: a7c4573db86a415285d274ca57dde659, State: Running, Role: LEADER
I20240502 01:55:51.501539 16884 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.502133 20614 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.501964 20293 consensus_queue.cc:237] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:51.530897 16358 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.531872 20689 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.536918 16224 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.537694 20689 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.539243 20614 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.539592 16075 catalog_manager.cc:5484] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 reported cstate change: term changed from 1 to 2, leader changed from 39059854b64045698ded1ba4e8d47304 (127.31.161.195) to a7c4573db86a415285d274ca57dde659 (127.31.161.201). New cstate: current_term: 2 leader_uuid: "a7c4573db86a415285d274ca57dde659" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:51.547034 16488 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.548151 16075 catalog_manager.cc:5484] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 reported cstate change: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:51.552772 16884 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.552731 17014 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.565369 18913 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.568693 18647 raft_consensus.cc:1203] T 297561b66dd7472ab817c2a23723565c P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 01:55:51.574088 18647 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.577136 20634 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.579689 16224 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.597532 16223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a81a0b7d47479e929d80041f34a3b1" candidate_uuid: "a7c4573db86a415285d274ca57dde659" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "cdd4d50250184223992325c86cb6c68e"
I20240502 01:55:51.597774 16223 raft_consensus.cc:3037] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:51.600589 16223 raft_consensus.cc:2454] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7c4573db86a415285d274ca57dde659 in term 2.
I20240502 01:55:51.613809 16357 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.622606 18911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a81a0b7d47479e929d80041f34a3b1" candidate_uuid: "a7c4573db86a415285d274ca57dde659" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "0982e15b4c74425cad8d7d642eb63316"
I20240502 01:55:51.622949 18911 raft_consensus.cc:3037] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:51.623749 16359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a81a0b7d47479e929d80041f34a3b1" candidate_uuid: "a7c4573db86a415285d274ca57dde659" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb"
I20240502 01:55:51.623977 16359 raft_consensus.cc:3037] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Advancing to term 2
I20240502 01:55:51.626152 18911 raft_consensus.cc:2454] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7c4573db86a415285d274ca57dde659 in term 2.
I20240502 01:55:51.626587 16359 raft_consensus.cc:2454] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7c4573db86a415285d274ca57dde659 in term 2.
I20240502 01:55:51.626318 18647 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.884672 16488 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:55:51.885321 20293 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.931234 20614 raft_consensus.cc:1050] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367: attempting to promote NON_VOTER 0982e15b4c74425cad8d7d642eb63316 to VOTER
I20240502 01:55:51.932327 20614 consensus_queue.cc:237] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:51.935443 18911 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.935456 18779 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.935467 16224 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.936096 20391 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.936226 16358 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.936264 16488 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.936442 20634 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.936790 20612 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.937144 20815 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.937888 20689 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.940485 17014 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.941296 20612 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.946589 18913 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316 [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:55:51.947471 20293 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.950179 17016 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:55:51.950735 20293 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.965504 20391 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.966159 16358 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.967446 18911 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.969661 16489 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.971050 16224 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:55:51.971639 20293 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.973769 16884 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:51.973923 18779 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.975368 17014 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.977293 20391 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:51.981139 16075 catalog_manager.cc:5484] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.981716 16223 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.986838 16620 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:55:51.986389 16884 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } attrs { replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:51.988346 20293 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:51.989486 16754 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:55:51.991688 20745 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:52.001379 16358 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 01:55:52.002424 20293 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:52.060654 16620 consensus_queue.cc:237] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:52.063338 18911 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:52.063580 18647 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:52.063931 20634 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:52.064209 20815 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:52.067586 16224 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:52.068075 20816 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:52.070472 20634 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.071177 16358 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:52.071733 20815 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:52.072806 16224 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.073357 16359 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.075650 17014 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:52.076073 16070 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cec84c633b0a4c4fb6e074c79432fe61 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:52.076258 20816 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:52.077489 16884 raft_consensus.cc:1261] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:52.078140 20816 consensus_queue.cc:1035] T cec84c633b0a4c4fb6e074c79432fe61 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:52.077936 17014 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.079777 16884 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.083550 18647 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.084141 16075 catalog_manager.cc:5484] T cec84c633b0a4c4fb6e074c79432fe61 P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.088225 18911 raft_consensus.cc:2932] T cec84c633b0a4c4fb6e074c79432fe61 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.091456 16471 tablet_service.cc:1514] Processing DeleteTablet for tablet cec84c633b0a4c4fb6e074c79432fe61 with delete_type TABLET_DATA_TOMBSTONED (TS 39059854b64045698ded1ba4e8d47304 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51446
I20240502 01:55:52.104818 20842 tablet_replica.cc:330] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304: stopping tablet replica
I20240502 01:55:52.105144 20842 raft_consensus.cc:2229] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:52.105439 20842 raft_consensus.cc:2258] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:52.107715 20842 ts_tablet_manager.cc:1875] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:52.115079 20842 ts_tablet_manager.cc:1888] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 01:55:52.115228 20842 log.cc:1196] T cec84c633b0a4c4fb6e074c79432fe61 P 39059854b64045698ded1ba4e8d47304: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/wals/cec84c633b0a4c4fb6e074c79432fe61
I20240502 01:55:52.116029 16067 catalog_manager.cc:4830] TS 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693): tablet cec84c633b0a4c4fb6e074c79432fe61 (table rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]) successfully deleted
I20240502 01:55:52.154508 20863 raft_consensus.cc:1050] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9: attempting to promote NON_VOTER a7c4573db86a415285d274ca57dde659 to VOTER
I20240502 01:55:52.155436 20863 consensus_queue.cc:237] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:52.158421 18647 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:52.158643 18779 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:52.159097 16620 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:52.159056 20526 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:52.159637 20526 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:52.159941 16884 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:52.160492 20153 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:52.165719 16489 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:52.170743 20153 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:52.171259 20153 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:52.186414 20526 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:52.191327 16621 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:52.202334 16359 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:52.202988 16489 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:52.203524 20863 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:52.205423 16359 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:52.211534 16224 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:52.212003 16081 catalog_manager.cc:5484] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:52.216234 20868 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:52.221109 18779 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:52.227408 16884 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:52.228648 16224 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:52.459465 18647 consensus_queue.cc:237] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:52.463424 16224 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:55:52.463719 18911 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316 [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:55:52.463719 16754 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:55:52.463981 17014 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:55:52.463981 16359 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:55:52.464406 20293 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:52.464795 20887 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:52.465361 16621 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 01:55:52.465575 20293 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:52.467402 20745 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:52.467995 20745 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:52.469355 20745 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:52.474546 16620 consensus_queue.cc:237] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:52.474196 20745 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.476295 16754 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.477818 18911 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:52.478184 17014 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.480577 18779 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:52.481297 20612 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:52.481719 16359 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:52.481827 20612 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:52.482362 20612 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:52.483237 16069 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 48a81a0b7d47479e929d80041f34a3b1 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:52.484586 16075 catalog_manager.cc:5484] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: config changed from index 9 to 11, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New cstate: current_term: 2 leader_uuid: "a7c4573db86a415285d274ca57dde659" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.487125 16225 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:52.488828 20815 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:52.489559 16358 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.489719 18913 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.493638 16471 tablet_service.cc:1514] Processing DeleteTablet for tablet 48a81a0b7d47479e929d80041f34a3b1 with delete_type TABLET_DATA_TOMBSTONED (TS 39059854b64045698ded1ba4e8d47304 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:51446
I20240502 01:55:52.494129 20842 tablet_replica.cc:330] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304: stopping tablet replica
I20240502 01:55:52.494359 20842 raft_consensus.cc:2229] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:52.494561 20842 raft_consensus.cc:2258] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:52.495450 20842 ts_tablet_manager.cc:1875] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:52.498440 17016 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:52.499115 16884 raft_consensus.cc:1261] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:52.499169 20816 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:52.498915 20634 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.499815 20614 consensus_queue.cc:1035] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:52.500746 17016 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.502542 16224 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.505177 20842 ts_tablet_manager.cc:1888] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 01:55:52.501655 16884 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.507886 18779 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.507872 18911 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.509770 20842 log.cc:1196] T 48a81a0b7d47479e929d80041f34a3b1 P 39059854b64045698ded1ba4e8d47304: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/wals/48a81a0b7d47479e929d80041f34a3b1
I20240502 01:55:52.510689 16067 catalog_manager.cc:4830] TS 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693): tablet 48a81a0b7d47479e929d80041f34a3b1 (table rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]) successfully deleted
I20240502 01:55:52.512197 16070 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bc9cc69a319d46b8940f4c7dc2e7e656 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:52.494179 16621 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.513648 16081 catalog_manager.cc:5484] T bc9cc69a319d46b8940f4c7dc2e7e656 P 703f6f043c6041a4b629933690580367 reported cstate change: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:52.520231 16471 tablet_service.cc:1514] Processing DeleteTablet for tablet bc9cc69a319d46b8940f4c7dc2e7e656 with delete_type TABLET_DATA_TOMBSTONED (TS 39059854b64045698ded1ba4e8d47304 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51446
I20240502 01:55:52.519716 16222 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.522320 20842 tablet_replica.cc:330] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304: stopping tablet replica
I20240502 01:55:52.522544 20842 raft_consensus.cc:2229] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:52.522894 20842 raft_consensus.cc:2258] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:52.523896 20842 ts_tablet_manager.cc:1875] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:52.530969 20842 ts_tablet_manager.cc:1888] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 01:55:52.531160 20842 log.cc:1196] T bc9cc69a319d46b8940f4c7dc2e7e656 P 39059854b64045698ded1ba4e8d47304: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-2/wal/wals/bc9cc69a319d46b8940f4c7dc2e7e656
I20240502 01:55:52.532815 16067 catalog_manager.cc:4830] TS 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693): tablet bc9cc69a319d46b8940f4c7dc2e7e656 (table rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]) successfully deleted
I20240502 01:55:52.533684 16359 raft_consensus.cc:2932] T bc9cc69a319d46b8940f4c7dc2e7e656 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 39059854b64045698ded1ba4e8d47304 (127.31.161.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:52.733726 18647 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:52.736120 17014 consensus_queue.cc:237] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:52.739701 18647 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:52.740342 16489 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:52.740478 16884 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:52.740402 20873 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:52.741119 20873 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:52.741535 20875 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:52.742007 18779 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:52.742893 16621 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:52.743641 20874 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:52.744407 20873 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:52.746047 16359 raft_consensus.cc:1261] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:52.746619 20875 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:52.748646 16621 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:52.748807 18779 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:52.749677 16884 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:52.753904 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 297561b66dd7472ab817c2a23723565c with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:52.754163 16082 catalog_manager.cc:5484] T 297561b66dd7472ab817c2a23723565c P 703f6f043c6041a4b629933690580367 reported cstate change: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:52.753243 20911 consensus_queue.cc:1035] T 297561b66dd7472ab817c2a23723565c P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.005s
I20240502 01:55:52.757273 16489 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:52.758186 16359 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:52.758258 18647 raft_consensus.cc:2932] T 297561b66dd7472ab817c2a23723565c P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:52.759790 16204 tablet_service.cc:1514] Processing DeleteTablet for tablet 297561b66dd7472ab817c2a23723565c with delete_type TABLET_DATA_TOMBSTONED (TS cdd4d50250184223992325c86cb6c68e not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:40596
I20240502 01:55:52.776216 20914 tablet_replica.cc:330] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e: stopping tablet replica
I20240502 01:55:52.776719 20914 raft_consensus.cc:2229] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:52.776988 20914 raft_consensus.cc:2258] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:52.778371 20914 ts_tablet_manager.cc:1875] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:52.783725 20914 ts_tablet_manager.cc:1888] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 01:55:52.783854 20914 log.cc:1196] T 297561b66dd7472ab817c2a23723565c P cdd4d50250184223992325c86cb6c68e: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-0/wal/wals/297561b66dd7472ab817c2a23723565c
I20240502 01:55:52.784584 16069 catalog_manager.cc:4830] TS cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953): tablet 297561b66dd7472ab817c2a23723565c (table rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]) successfully deleted
I20240502 01:55:52.920691 16471 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:52.925182 18893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:52.948236 16602 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:52.973924 16864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:52.979048 18626 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:52.981616 16995 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:53.060230 18760 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:53.097816 16733 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:53.106264 16204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:53.122048 16339 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:53.357861 16471 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:53.384857 16995 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:53.397329 16602 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:53.401957 18626 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:53.406996 16864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:53.407701 18893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:53.429391 18760 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:53.448765 16204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:53.490235 16339 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:53.531366 16733 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:53.681286 16754 consensus_queue.cc:237] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: NON_VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: true } }
I20240502 01:55:53.682487 16489 consensus_queue.cc:237] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } }
I20240502 01:55:53.684494 17014 consensus_queue.cc:237] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:53.688413 18779 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240502 01:55:53.688568 16426 consensus_peers.cc:485] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 -> Peer 703f6f043c6041a4b629933690580367 (127.31.161.196:40425): Couldn't send request to peer 703f6f043c6041a4b629933690580367. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:53.688568 16751 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:53.689015 18913 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:53.689173 21007 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:53.689519 16222 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:53.689723 21040 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:53.690158 16884 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:53.690218 21007 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:53.690706 17016 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:53.691455 19782 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:53.692611 18647 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:53.692998 16489 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:53.693346 18911 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240502 01:55:53.693470 16689 consensus_peers.cc:485] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad -> Peer cdd4d50250184223992325c86cb6c68e (127.31.161.193:41953): Couldn't send request to peer cdd4d50250184223992325c86cb6c68e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:53.693535 16359 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:53.693672 20956 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:53.694195 16884 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:53.694267 20956 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:53.694802 20956 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:53.694937 17016 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:53.695084 20715 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:53.695633 20715 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:53.697008 20956 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:53.697829 19782 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:53.698361 19782 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:53.699690 16358 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:53.700001 16224 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:53.700915 18780 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:53.700978 20911 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:53.701479 20913 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:53.702924 16884 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:53.703346 20913 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:53.704084 16751 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 01:55:53.704098 16951 consensus_peers.cc:485] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 -> Peer a7c4573db86a415285d274ca57dde659 (127.31.161.201:44737): Couldn't send request to peer a7c4573db86a415285d274ca57dde659. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bab1fc6db80c4e53a4f0cacccca74096. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:53.705098 16488 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:53.705032 20913 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:53.713853 20911 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:53.714407 20911 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:53.718652 20578 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:53.722185 18778 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:53.725724 16222 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:53.732296 16882 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:53.739496 16082 catalog_manager.cc:5484] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a reported cstate change: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:53.744303 16752 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:53.750102 18913 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:53.789448 20913 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:53.795768 16488 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:53.797032 16882 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:53.805590 16224 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:53.814858 20956 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: NON_VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: true } } }
I20240502 01:55:53.817901 18646 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: NON_VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: true } } }
I20240502 01:55:53.831007 16081 catalog_manager.cc:5484] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659 reported cstate change: config changed from index 10 to 11, NON_VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) added. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: NON_VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: true } } }
I20240502 01:55:53.832800 16489 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: NON_VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: true } } }
I20240502 01:55:53.870347 16359 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:53.879957 18778 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
W20240502 01:55:53.884929 16688 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.200:38559, user_credentials={real_user=slave}} blocked reactor thread for 190149us
I20240502 01:55:53.898226 17016 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:53.923857 18913 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: NON_VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: true } } }
I20240502 01:55:53.939977 17016 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: NON_VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: true } } }
I20240502 01:55:53.942018 16751 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:53.943361 16882 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: NON_VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: true } } }
I20240502 01:55:53.948037 16074 catalog_manager.cc:5484] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index 7 to 8, NON_VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) added. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } } }
I20240502 01:55:53.946835 16359 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER cdd4d50250184223992325c86cb6c68e (127.31.161.193) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: NON_VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: true } } }
I20240502 01:55:53.959726 18647 consensus_queue.cc:237] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 9, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: true } }
I20240502 01:55:53.973546 18913 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316 [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:55:53.974886 20887 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:53.977733 16359 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:55:53.979087 20887 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:53.981384 16222 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:55:53.981943 20887 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:53.986840 17016 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:55:53.987221 16620 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:55:53.987870 20887 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:53.988255 16754 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 01:55:53.988519 20887 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:53.992642 21093 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 01:55:53.995625 18580 consensus_peers.cc:485] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 -> Peer 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121): Couldn't send request to peer 6b9f9aaa1c86420190af75270914449a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:53.998804 16751 consensus_queue.cc:237] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
W20240502 01:55:54.013377 16688 consensus_peers.cc:485] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad -> Peer aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202:43977): Couldn't send request to peer aa7ec9abef1747e5a9e2404cee265c20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a4e2e002a7d441398f3da0ec650b8d2. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:54.013931 18913 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.014477 21071 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:54.016469 16489 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.017300 16222 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.017446 17015 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.018362 21071 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:54.018757 16882 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.018883 21043 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:54.020020 16358 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.020576 20956 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:54.021121 20956 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:54.021587 20956 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:54.058246 20887 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: true } } }
I20240502 01:55:54.060322 16621 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: true } } }
I20240502 01:55:54.067514 16079 catalog_manager.cc:5484] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 reported cstate change: config changed from index 11 to 12, NON_VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) added. New cstate: current_term: 2 leader_uuid: "a7c4573db86a415285d274ca57dde659" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: true } } }
I20240502 01:55:54.072227 18912 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: true } } }
I20240502 01:55:54.073482 21071 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:54.086035 16224 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: true } } }
I20240502 01:55:54.092674 16079 catalog_manager.cc:5484] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:54.092964 16882 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:54.098073 17015 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:54.098387 16489 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:54.114217 21110 ts_tablet_manager.cc:902] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e: Initiating tablet copy from peer ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825)
I20240502 01:55:54.117313 16751 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: true } } }
I20240502 01:55:54.126317 18912 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:54.137269 16222 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:54.143033 17015 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: true } } }
I20240502 01:55:54.143858 16357 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: true } } }
I20240502 01:55:54.140311 16358 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:54.160326 21110 tablet_copy_client.cc:308] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.197:46825
I20240502 01:55:54.161367 16774 tablet_copy_service.cc:140] P ed2e7e4188d0400b8689e28bbf5096ad: Received BeginTabletCopySession request for tablet 512eab02225d4544a342811f0a7359aa from peer cdd4d50250184223992325c86cb6c68e ({username='slave'} at 127.31.161.193:41093)
I20240502 01:55:54.161532 16774 tablet_copy_service.cc:161] P ed2e7e4188d0400b8689e28bbf5096ad: Beginning new tablet copy session on tablet 512eab02225d4544a342811f0a7359aa from peer cdd4d50250184223992325c86cb6c68e at {username='slave'} at 127.31.161.193:41093: session id = cdd4d50250184223992325c86cb6c68e-512eab02225d4544a342811f0a7359aa
I20240502 01:55:54.163754 16774 tablet_copy_source_session.cc:205] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:54.164938 16882 consensus_queue.cc:237] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:54.165513 21110 ts_tablet_manager.cc:1875] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:55:54.168589 18647 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.169760 21016 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 01:55:54.171933 16819 consensus_peers.cc:485] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a -> Peer 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200:38559): Couldn't send request to peer 0982e15b4c74425cad8d7d642eb63316. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4da77826275d42b2b9e280c732575cfd. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:54.180377 21110 ts_tablet_manager.cc:1888] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240502 01:55:54.180718 21110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 512eab02225d4544a342811f0a7359aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:54.185463 16751 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.186486 16489 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.185051 17015 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.187893 21016 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:54.188465 21016 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:54.188979 21016 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:54.193202 21110 tablet_copy_client.cc:783] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:54.193533 21110 tablet_copy_client.cc:647] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:54.199234 21016 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:54.200634 16751 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:54.205226 21110 tablet_copy_client.cc:523] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:54.209406 21110 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e: Bootstrap starting.
I20240502 01:55:54.213780 16489 consensus_queue.cc:237] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } }
I20240502 01:55:54.217459 16075 catalog_manager.cc:5484] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New cstate: current_term: 1 leader_uuid: "6b9f9aaa1c86420190af75270914449a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:54.225711 18647 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.226308 17015 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.226337 19782 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:54.227401 16358 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.227815 16222 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.231305 21007 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 01:55:54.231402 16423 consensus_peers.cc:485] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 -> Peer ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825): Couldn't send request to peer ed2e7e4188d0400b8689e28bbf5096ad. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:54.231891 19782 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:54.232347 16882 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.232684 16621 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.232880 19782 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:54.234112 21062 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:54.234709 21062 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:54.263126 16224 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.263289 16359 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.264148 21127 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:54.263935 17016 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:54.268990 19782 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:54.272457 21134 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:54.273452 16884 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:54.274437 18647 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:54.278658 16620 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:54.283932 16079 catalog_manager.cc:5484] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659 reported cstate change: config changed from index 7 to 8, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:54.284793 21145 ts_tablet_manager.cc:902] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367: Initiating tablet copy from peer 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:54.285938 16359 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:54.290937 16224 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:54.303370 21110 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:54.303840 21110 tablet_bootstrap.cc:492] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e: Bootstrap complete.
I20240502 01:55:54.304168 21110 ts_tablet_manager.cc:1367] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e: Time spent bootstrapping tablet: real 0.095s user 0.053s sys 0.008s
I20240502 01:55:54.304987 21110 raft_consensus.cc:348] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: NON_VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: true } }
I20240502 01:55:54.305363 21110 raft_consensus.cc:729] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cdd4d50250184223992325c86cb6c68e, State: Initialized, Role: LEARNER
I20240502 01:55:54.305639 21110 consensus_queue.cc:260] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: NON_VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: true } }
I20240502 01:55:54.306440 21110 ts_tablet_manager.cc:1398] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 01:55:54.307229 16774 tablet_copy_service.cc:342] P ed2e7e4188d0400b8689e28bbf5096ad: Request end of tablet copy session cdd4d50250184223992325c86cb6c68e-512eab02225d4544a342811f0a7359aa received from {username='slave'} at 127.31.161.193:41093
I20240502 01:55:54.307384 16774 tablet_copy_service.cc:433] P ed2e7e4188d0400b8689e28bbf5096ad: ending tablet copy session cdd4d50250184223992325c86cb6c68e-512eab02225d4544a342811f0a7359aa on tablet 512eab02225d4544a342811f0a7359aa with peer cdd4d50250184223992325c86cb6c68e
I20240502 01:55:54.311893 21145 tablet_copy_client.cc:308] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:35693
I20240502 01:55:54.312703 16511 tablet_copy_service.cc:140] P 39059854b64045698ded1ba4e8d47304: Received BeginTabletCopySession request for tablet 1fa6224f34ef4bc58797f6cf801ff6da from peer 703f6f043c6041a4b629933690580367 ({username='slave'} at 127.31.161.196:58449)
I20240502 01:55:54.312888 16511 tablet_copy_service.cc:161] P 39059854b64045698ded1ba4e8d47304: Beginning new tablet copy session on tablet 1fa6224f34ef4bc58797f6cf801ff6da from peer 703f6f043c6041a4b629933690580367 at {username='slave'} at 127.31.161.196:58449: session id = 703f6f043c6041a4b629933690580367-1fa6224f34ef4bc58797f6cf801ff6da
I20240502 01:55:54.315709 16511 tablet_copy_source_session.cc:205] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:54.319154 16224 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:54.321022 21145 ts_tablet_manager.cc:1875] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:55:54.328946 21149 ts_tablet_manager.cc:902] T bab1fc6db80c4e53a4f0cacccca74096 P a7c4573db86a415285d274ca57dde659: Initiating tablet copy from peer a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:54.330243 16358 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:54.340123 16489 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:54.341975 17016 consensus_queue.cc:237] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } }
W20240502 01:55:54.347549 16952 consensus_peers.cc:485] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 -> Peer 703f6f043c6041a4b629933690580367 (127.31.161.196:40425): Couldn't send request to peer 703f6f043c6041a4b629933690580367. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:54.348234 16358 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:54.348438 16751 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:54.348783 20913 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.350920 20913 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.353410 18911 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:54.354445 18646 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:54.355149 20913 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.362221 16884 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:54.363054 20913 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.362957 17015 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:54.365764 21145 ts_tablet_manager.cc:1888] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 01:55:54.366891 21145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa6224f34ef4bc58797f6cf801ff6da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:54.367511 16884 consensus_queue.cc:237] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:54.371443 21050 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.374301 18646 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:54.375142 21127 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.383966 16359 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:54.384475 16620 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:54.385126 16489 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:54.385191 21127 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.385777 21127 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.386317 17016 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:54.386277 21134 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.386941 21134 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.389660 21145 tablet_copy_client.cc:783] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:54.389390 21050 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:54.390064 21145 tablet_copy_client.cc:647] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:54.392067 21145 tablet_copy_client.cc:523] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:54.397138 21145 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:54.397845 16358 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:54.398137 18646 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:54.398568 16754 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:54.404299 16075 catalog_manager.cc:5484] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:54.424391 16882 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:54.425992 18647 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } } }
I20240502 01:55:54.427348 16222 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:54.428500 18913 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:54.433519 20911 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.440385 21134 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:54.447918 16488 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:54.449445 18647 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:54.451571 16621 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:54.455261 16359 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:54.458989 16075 catalog_manager.cc:5484] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index 10 to 11, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New cstate: current_term: 1 leader_uuid: "6b9f9aaa1c86420190af75270914449a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:54.469743 17016 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
I20240502 01:55:54.470732 18913 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:54.471879 16222 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:54.476224 21016 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.496021 18913 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } } }
W20240502 01:55:54.507982 16819 consensus_peers.cc:485] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a -> Peer aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202:43977): Couldn't send request to peer aa7ec9abef1747e5a9e2404cee265c20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 453dff7b204a431592ac15c058db0493. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:54.510592 16620 consensus_queue.cc:237] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } }
W20240502 01:55:54.518431 16555 consensus_peers.cc:485] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 -> Peer ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825): Couldn't send request to peer ed2e7e4188d0400b8689e28bbf5096ad. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:54.518493 18780 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.519475 20634 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:54.519579 16489 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.520421 20689 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:54.521939 16222 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.523948 20634 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:54.525396 16884 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.526804 20634 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:54.526906 17016 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.527689 20689 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:54.531283 16359 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 01:55:54.531965 20634 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 01:55:54.548806 20634 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:54.552911 16883 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:54.559913 16075 catalog_manager.cc:5484] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a reported cstate change: config changed from index 7 to 8, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:54.567708 16489 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:54.569458 18780 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:54.579372 21149 tablet_copy_client.cc:308] T bab1fc6db80c4e53a4f0cacccca74096 P a7c4573db86a415285d274ca57dde659: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.199:46837
I20240502 01:55:54.580111 17036 tablet_copy_service.cc:140] P a6575ebf978a4631a13ba5c243c3c4f9: Received BeginTabletCopySession request for tablet bab1fc6db80c4e53a4f0cacccca74096 from peer a7c4573db86a415285d274ca57dde659 ({username='slave'} at 127.31.161.201:32955)
I20240502 01:55:54.580284 17036 tablet_copy_service.cc:161] P a6575ebf978a4631a13ba5c243c3c4f9: Beginning new tablet copy session on tablet bab1fc6db80c4e53a4f0cacccca74096 from peer a7c4573db86a415285d274ca57dde659 at {username='slave'} at 127.31.161.201:32955: session id = a7c4573db86a415285d274ca57dde659-bab1fc6db80c4e53a4f0cacccca74096
I20240502 01:55:54.583181 17036 tablet_copy_source_session.cc:205] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:54.584450 21149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bab1fc6db80c4e53a4f0cacccca74096. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:54.595939 21145 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:54.596452 21145 tablet_bootstrap.cc:492] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367: Bootstrap complete.
I20240502 01:55:54.596769 21145 ts_tablet_manager.cc:1367] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.200s user 0.047s sys 0.008s
I20240502 01:55:54.597592 21145 raft_consensus.cc:348] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } }
I20240502 01:55:54.597920 21145 raft_consensus.cc:729] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: LEARNER
I20240502 01:55:54.598131 21145 consensus_queue.cc:260] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } }
I20240502 01:55:54.598897 21149 tablet_copy_client.cc:783] T bab1fc6db80c4e53a4f0cacccca74096 P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:54.599149 21149 tablet_copy_client.cc:647] T bab1fc6db80c4e53a4f0cacccca74096 P a7c4573db86a415285d274ca57dde659: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:54.601796 21149 tablet_copy_client.cc:523] T bab1fc6db80c4e53a4f0cacccca74096 P a7c4573db86a415285d274ca57dde659: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:54.604976 21145 ts_tablet_manager.cc:1398] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20240502 01:55:54.605254 21149 tablet_bootstrap.cc:492] T bab1fc6db80c4e53a4f0cacccca74096 P a7c4573db86a415285d274ca57dde659: Bootstrap starting.
I20240502 01:55:54.605727 16511 tablet_copy_service.cc:342] P 39059854b64045698ded1ba4e8d47304: Request end of tablet copy session 703f6f043c6041a4b629933690580367-1fa6224f34ef4bc58797f6cf801ff6da received from {username='slave'} at 127.31.161.196:58449
I20240502 01:55:54.605885 16511 tablet_copy_service.cc:433] P 39059854b64045698ded1ba4e8d47304: ending tablet copy session 703f6f043c6041a4b629933690580367-1fa6224f34ef4bc58797f6cf801ff6da on tablet 1fa6224f34ef4bc58797f6cf801ff6da with peer 703f6f043c6041a4b629933690580367
I20240502 01:55:54.611081 21216 ts_tablet_manager.cc:902] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad: Initiating tablet copy from peer 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:54.623484 16224 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:54.630008 16358 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:54.631412 16489 consensus_queue.cc:237] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } }
I20240502 01:55:54.646611 18779 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:54.647326 21062 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.656599 18911 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:54.657202 21062 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.658133 17015 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:54.663838 16224 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:54.664520 21062 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.667757 21222 ts_tablet_manager.cc:902] T 3a4e2e002a7d441398f3da0ec650b8d2 P aa7ec9abef1747e5a9e2404cee265c20: Initiating tablet copy from peer ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825)
I20240502 01:55:54.668782 21218 ts_tablet_manager.cc:902] T 4da77826275d42b2b9e280c732575cfd P 0982e15b4c74425cad8d7d642eb63316: Initiating tablet copy from peer 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121)
I20240502 01:55:54.675369 17016 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:54.676199 20578 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.677513 16884 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240502 01:55:54.678195 16426 consensus_peers.cc:485] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 -> Peer 703f6f043c6041a4b629933690580367 (127.31.161.196:40425): Couldn't send request to peer 703f6f043c6041a4b629933690580367. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:54.678191 21062 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.679394 16754 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:54.680001 20578 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.685232 21198 ts_tablet_manager.cc:902] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: Initiating tablet copy from peer a7c4573db86a415285d274ca57dde659 (127.31.161.201:44737)
I20240502 01:55:54.690445 16620 raft_consensus.cc:1203] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240502 01:55:54.696601 21216 tablet_copy_client.cc:308] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:35693
I20240502 01:55:54.697095 21064 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:54.700743 16222 raft_consensus.cc:1203] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
W20240502 01:55:54.701162 21223 log.cc:924] Time spent T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20240502 01:55:54.702294 16511 tablet_copy_service.cc:140] P 39059854b64045698ded1ba4e8d47304: Received BeginTabletCopySession request for tablet 479fcadbde0449f4bfeef0bd2f82ebaf from peer ed2e7e4188d0400b8689e28bbf5096ad ({username='slave'} at 127.31.161.197:49847)
I20240502 01:55:54.702502 16511 tablet_copy_service.cc:161] P 39059854b64045698ded1ba4e8d47304: Beginning new tablet copy session on tablet 479fcadbde0449f4bfeef0bd2f82ebaf from peer ed2e7e4188d0400b8689e28bbf5096ad at {username='slave'} at 127.31.161.197:49847: session id = ed2e7e4188d0400b8689e28bbf5096ad-479fcadbde0449f4bfeef0bd2f82ebaf
I20240502 01:55:54.702800 16225 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:54.711161 16511 tablet_copy_source_session.cc:205] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:54.714411 21216 ts_tablet_manager.cc:1875] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:55:54.719666 18779 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:54.726907 16081 catalog_manager.cc:5484] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20 reported cstate change: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:54.742211 21216 ts_tablet_manager.cc:1888] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 01:55:54.742549 21216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 479fcadbde0449f4bfeef0bd2f82ebaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:54.743405 21218 tablet_copy_client.cc:308] T 4da77826275d42b2b9e280c732575cfd P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.198:38121
I20240502 01:55:54.746677 16905 tablet_copy_service.cc:140] P 6b9f9aaa1c86420190af75270914449a: Received BeginTabletCopySession request for tablet 4da77826275d42b2b9e280c732575cfd from peer 0982e15b4c74425cad8d7d642eb63316 ({username='slave'} at 127.31.161.200:50837)
I20240502 01:55:54.746876 16905 tablet_copy_service.cc:161] P 6b9f9aaa1c86420190af75270914449a: Beginning new tablet copy session on tablet 4da77826275d42b2b9e280c732575cfd from peer 0982e15b4c74425cad8d7d642eb63316 at {username='slave'} at 127.31.161.200:50837: session id = 0982e15b4c74425cad8d7d642eb63316-4da77826275d42b2b9e280c732575cfd
I20240502 01:55:54.749663 16905 tablet_copy_source_session.cc:205] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:54.749504 17016 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:54.755429 21218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da77826275d42b2b9e280c732575cfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:54.757265 18913 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:54.773217 16754 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:54.782963 21245 ts_tablet_manager.cc:902] T 453dff7b204a431592ac15c058db0493 P aa7ec9abef1747e5a9e2404cee265c20: Initiating tablet copy from peer 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121)
I20240502 01:55:54.808727 21222 tablet_copy_client.cc:308] T 3a4e2e002a7d441398f3da0ec650b8d2 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.197:46825
I20240502 01:55:54.809592 16884 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } } }
I20240502 01:55:54.810537 16774 tablet_copy_service.cc:140] P ed2e7e4188d0400b8689e28bbf5096ad: Received BeginTabletCopySession request for tablet 3a4e2e002a7d441398f3da0ec650b8d2 from peer aa7ec9abef1747e5a9e2404cee265c20 ({username='slave'} at 127.31.161.202:42599)
I20240502 01:55:54.810714 16774 tablet_copy_service.cc:161] P ed2e7e4188d0400b8689e28bbf5096ad: Beginning new tablet copy session on tablet 3a4e2e002a7d441398f3da0ec650b8d2 from peer aa7ec9abef1747e5a9e2404cee265c20 at {username='slave'} at 127.31.161.202:42599: session id = aa7ec9abef1747e5a9e2404cee265c20-3a4e2e002a7d441398f3da0ec650b8d2
I20240502 01:55:54.811244 21216 tablet_copy_client.cc:783] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:54.812444 21216 tablet_copy_client.cc:647] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:54.813652 16774 tablet_copy_source_session.cc:205] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:54.819801 21218 tablet_copy_client.cc:783] T 4da77826275d42b2b9e280c732575cfd P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:54.820237 21218 tablet_copy_client.cc:647] T 4da77826275d42b2b9e280c732575cfd P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:54.824069 21222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4e2e002a7d441398f3da0ec650b8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:54.835836 21218 tablet_copy_client.cc:523] T 4da77826275d42b2b9e280c732575cfd P 0982e15b4c74425cad8d7d642eb63316: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:54.843097 21216 tablet_copy_client.cc:523] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:54.846910 21216 tablet_bootstrap.cc:492] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:54.851629 21245 tablet_copy_client.cc:308] T 453dff7b204a431592ac15c058db0493 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.198:38121
I20240502 01:55:54.854748 21198 tablet_copy_client.cc:308] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.201:44737
I20240502 01:55:54.855901 18667 tablet_copy_service.cc:140] P a7c4573db86a415285d274ca57dde659: Received BeginTabletCopySession request for tablet 48a81a0b7d47479e929d80041f34a3b1 from peer 6b9f9aaa1c86420190af75270914449a ({username='slave'} at 127.31.161.198:55601)
I20240502 01:55:54.856122 18667 tablet_copy_service.cc:161] P a7c4573db86a415285d274ca57dde659: Beginning new tablet copy session on tablet 48a81a0b7d47479e929d80041f34a3b1 from peer 6b9f9aaa1c86420190af75270914449a at {username='slave'} at 127.31.161.198:55601: session id = 6b9f9aaa1c86420190af75270914449a-48a81a0b7d47479e929d80041f34a3b1
I20240502 01:55:54.860989 18667 tablet_copy_source_session.cc:205] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:54.866039 21198 ts_tablet_manager.cc:1875] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:55:54.874400 16905 tablet_copy_service.cc:140] P 6b9f9aaa1c86420190af75270914449a: Received BeginTabletCopySession request for tablet 453dff7b204a431592ac15c058db0493 from peer aa7ec9abef1747e5a9e2404cee265c20 ({username='slave'} at 127.31.161.202:52497)
I20240502 01:55:54.874565 16905 tablet_copy_service.cc:161] P 6b9f9aaa1c86420190af75270914449a: Beginning new tablet copy session on tablet 453dff7b204a431592ac15c058db0493 from peer aa7ec9abef1747e5a9e2404cee265c20 at {username='slave'} at 127.31.161.202:52497: session id = aa7ec9abef1747e5a9e2404cee265c20-453dff7b204a431592ac15c058db0493
I20240502 01:55:54.877552 16905 tablet_copy_source_session.cc:205] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:54.878005 21218 tablet_bootstrap.cc:492] T 4da77826275d42b2b9e280c732575cfd P 0982e15b4c74425cad8d7d642eb63316: Bootstrap starting.
I20240502 01:55:54.878986 21245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 453dff7b204a431592ac15c058db0493. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:54.885447 21245 tablet_copy_client.cc:783] T 453dff7b204a431592ac15c058db0493 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:54.885752 21245 tablet_copy_client.cc:647] T 453dff7b204a431592ac15c058db0493 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:54.887472 21245 tablet_copy_client.cc:523] T 453dff7b204a431592ac15c058db0493 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:54.890667 21245 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap starting.
I20240502 01:55:54.898443 21222 tablet_copy_client.cc:783] T 3a4e2e002a7d441398f3da0ec650b8d2 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:54.898758 21222 tablet_copy_client.cc:647] T 3a4e2e002a7d441398f3da0ec650b8d2 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:54.905154 21222 tablet_copy_client.cc:523] T 3a4e2e002a7d441398f3da0ec650b8d2 P aa7ec9abef1747e5a9e2404cee265c20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:54.914191 21149 tablet_bootstrap.cc:492] T bab1fc6db80c4e53a4f0cacccca74096 P a7c4573db86a415285d274ca57dde659: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:54.914687 21149 tablet_bootstrap.cc:492] T bab1fc6db80c4e53a4f0cacccca74096 P a7c4573db86a415285d274ca57dde659: Bootstrap complete.
I20240502 01:55:54.914968 21149 ts_tablet_manager.cc:1367] T bab1fc6db80c4e53a4f0cacccca74096 P a7c4573db86a415285d274ca57dde659: Time spent bootstrapping tablet: real 0.310s user 0.093s sys 0.044s
I20240502 01:55:54.915488 16489 consensus_queue.cc:237] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } }
I20240502 01:55:54.915768 21149 raft_consensus.cc:348] T bab1fc6db80c4e53a4f0cacccca74096 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:54.916124 21149 raft_consensus.cc:729] T bab1fc6db80c4e53a4f0cacccca74096 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a7c4573db86a415285d274ca57dde659, State: Initialized, Role: LEARNER
I20240502 01:55:54.916338 21149 consensus_queue.cc:260] T bab1fc6db80c4e53a4f0cacccca74096 P a7c4573db86a415285d274ca57dde659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: NON_VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: true } }
I20240502 01:55:54.917169 21149 ts_tablet_manager.cc:1398] T bab1fc6db80c4e53a4f0cacccca74096 P a7c4573db86a415285d274ca57dde659: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 01:55:54.917879 17036 tablet_copy_service.cc:342] P a6575ebf978a4631a13ba5c243c3c4f9: Request end of tablet copy session a7c4573db86a415285d274ca57dde659-bab1fc6db80c4e53a4f0cacccca74096 received from {username='slave'} at 127.31.161.201:32955
I20240502 01:55:54.918030 17036 tablet_copy_service.cc:433] P a6575ebf978a4631a13ba5c243c3c4f9: ending tablet copy session a7c4573db86a415285d274ca57dde659-bab1fc6db80c4e53a4f0cacccca74096 on tablet bab1fc6db80c4e53a4f0cacccca74096 with peer a7c4573db86a415285d274ca57dde659
I20240502 01:55:54.920032 21198 ts_tablet_manager.cc:1888] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 01:55:54.921582 21222 tablet_bootstrap.cc:492] T 3a4e2e002a7d441398f3da0ec650b8d2 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap starting.
I20240502 01:55:54.925424 18647 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240502 01:55:54.925443 16423 consensus_peers.cc:485] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 -> Peer ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825): Couldn't send request to peer ed2e7e4188d0400b8689e28bbf5096ad. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 01:55:54.926066 21062 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.927511 17016 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:54.928215 21062 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.932149 16620 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:54.933598 21064 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.938493 21198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a81a0b7d47479e929d80041f34a3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:54.926461 18779 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:54.939249 21064 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.945407 21145 ts_tablet_manager.cc:902] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367: Initiating tablet copy from peer a6575ebf978a4631a13ba5c243c3c4f9 (127.31.161.199:46837)
I20240502 01:55:54.946738 16359 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:54.945699 16882 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 01:55:54.947392 21064 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.949492 21064 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 01:55:54.970508 21198 tablet_copy_client.cc:783] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:54.971134 21198 tablet_copy_client.cc:647] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:54.975941 21198 tablet_copy_client.cc:523] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:55.006150 21198 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: Bootstrap starting.
I20240502 01:55:55.014307 21062 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:55.019021 16358 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:55.029587 16075 catalog_manager.cc:5484] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index 10 to 11, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:55.037606 16620 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:55.042831 16882 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:55.065064 17015 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:55.065451 18779 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:55.068899 21288 ts_tablet_manager.cc:902] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad: Initiating tablet copy from peer 703f6f043c6041a4b629933690580367 (127.31.161.196:40425)
I20240502 01:55:55.086864 21145 tablet_copy_client.cc:308] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.199:46837
I20240502 01:55:55.087507 17036 tablet_copy_service.cc:140] P a6575ebf978a4631a13ba5c243c3c4f9: Received BeginTabletCopySession request for tablet a8f7e3c8b6d6434db2a99d7139599a5d from peer 703f6f043c6041a4b629933690580367 ({username='slave'} at 127.31.161.196:37261)
I20240502 01:55:55.087693 17036 tablet_copy_service.cc:161] P a6575ebf978a4631a13ba5c243c3c4f9: Beginning new tablet copy session on tablet a8f7e3c8b6d6434db2a99d7139599a5d from peer 703f6f043c6041a4b629933690580367 at {username='slave'} at 127.31.161.196:37261: session id = 703f6f043c6041a4b629933690580367-a8f7e3c8b6d6434db2a99d7139599a5d
I20240502 01:55:55.087836 21245 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:55.088213 21245 tablet_bootstrap.cc:492] T 453dff7b204a431592ac15c058db0493 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap complete.
I20240502 01:55:55.089677 21245 ts_tablet_manager.cc:1367] T 453dff7b204a431592ac15c058db0493 P aa7ec9abef1747e5a9e2404cee265c20: Time spent bootstrapping tablet: real 0.198s user 0.045s sys 0.017s
I20240502 01:55:55.090487 17036 tablet_copy_source_session.cc:205] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:55.092947 21145 ts_tablet_manager.cc:1875] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:55:55.099568 21145 ts_tablet_manager.cc:1888] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 01:55:55.099843 21145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8f7e3c8b6d6434db2a99d7139599a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:55.100950 21245 raft_consensus.cc:348] T 453dff7b204a431592ac15c058db0493 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:55.101284 21245 raft_consensus.cc:729] T 453dff7b204a431592ac15c058db0493 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa7ec9abef1747e5a9e2404cee265c20, State: Initialized, Role: LEARNER
I20240502 01:55:55.102077 21299 ts_tablet_manager.cc:902] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367: Initiating tablet copy from peer 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:55.101506 21245 consensus_queue.cc:260] T 453dff7b204a431592ac15c058db0493 P aa7ec9abef1747e5a9e2404cee265c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:55.103237 21245 ts_tablet_manager.cc:1398] T 453dff7b204a431592ac15c058db0493 P aa7ec9abef1747e5a9e2404cee265c20: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 01:55:55.114084 16905 tablet_copy_service.cc:342] P 6b9f9aaa1c86420190af75270914449a: Request end of tablet copy session aa7ec9abef1747e5a9e2404cee265c20-453dff7b204a431592ac15c058db0493 received from {username='slave'} at 127.31.161.202:52497
I20240502 01:55:55.114252 16905 tablet_copy_service.cc:433] P 6b9f9aaa1c86420190af75270914449a: ending tablet copy session aa7ec9abef1747e5a9e2404cee265c20-453dff7b204a431592ac15c058db0493 on tablet 453dff7b204a431592ac15c058db0493 with peer aa7ec9abef1747e5a9e2404cee265c20
I20240502 01:55:55.115700 21043 raft_consensus.cc:1050] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad: attempting to promote NON_VOTER cdd4d50250184223992325c86cb6c68e to VOTER
I20240502 01:55:55.116868 21043 consensus_queue.cc:237] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: false } }
I20240502 01:55:55.120304 21216 tablet_bootstrap.cc:492] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:55.124487 16489 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:55.124612 16359 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:55.124679 18913 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:55.124840 21145 tablet_copy_client.cc:783] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:55.125248 21145 tablet_copy_client.cc:647] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:55.125258 18646 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:55.125813 16224 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 LEARNER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:55.127429 21145 tablet_copy_client.cc:523] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:55.121634 21288 tablet_copy_client.cc:308] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.196:40425
I20240502 01:55:55.125131 21071 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:55.134240 21071 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.004s
I20240502 01:55:55.134653 20715 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:55.135300 20715 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:55.137686 16642 tablet_copy_service.cc:140] P 703f6f043c6041a4b629933690580367: Received BeginTabletCopySession request for tablet f53b466fba6a4655a63a559b0ea56f77 from peer ed2e7e4188d0400b8689e28bbf5096ad ({username='slave'} at 127.31.161.197:37875)
I20240502 01:55:55.137887 16642 tablet_copy_service.cc:161] P 703f6f043c6041a4b629933690580367: Beginning new tablet copy session on tablet f53b466fba6a4655a63a559b0ea56f77 from peer ed2e7e4188d0400b8689e28bbf5096ad at {username='slave'} at 127.31.161.197:37875: session id = ed2e7e4188d0400b8689e28bbf5096ad-f53b466fba6a4655a63a559b0ea56f77
I20240502 01:55:55.140924 16642 tablet_copy_source_session.cc:205] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:55.146139 21216 tablet_bootstrap.cc:492] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap complete.
I20240502 01:55:55.146471 21216 ts_tablet_manager.cc:1367] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.300s user 0.047s sys 0.008s
I20240502 01:55:55.148919 21288 ts_tablet_manager.cc:1875] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:55:55.155699 21288 ts_tablet_manager.cc:1888] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 01:55:55.156003 21288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f53b466fba6a4655a63a559b0ea56f77. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:55.160733 17016 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:55.165172 21299 tablet_copy_client.cc:308] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:35693
I20240502 01:55:55.165962 16511 tablet_copy_service.cc:140] P 39059854b64045698ded1ba4e8d47304: Received BeginTabletCopySession request for tablet 18e676bcd8b04f18a6814d6ebbd1b488 from peer 703f6f043c6041a4b629933690580367 ({username='slave'} at 127.31.161.196:58449)
I20240502 01:55:55.166105 16511 tablet_copy_service.cc:161] P 39059854b64045698ded1ba4e8d47304: Beginning new tablet copy session on tablet 18e676bcd8b04f18a6814d6ebbd1b488 from peer 703f6f043c6041a4b629933690580367 at {username='slave'} at 127.31.161.196:58449: session id = 703f6f043c6041a4b629933690580367-18e676bcd8b04f18a6814d6ebbd1b488
I20240502 01:55:55.170568 16511 tablet_copy_source_session.cc:205] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:55.171831 21299 ts_tablet_manager.cc:1875] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:55:55.173307 21071 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20240502 01:55:55.175138 20715 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:55.161358 21216 raft_consensus.cc:348] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } }
I20240502 01:55:55.178617 21216 raft_consensus.cc:729] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: LEARNER
I20240502 01:55:55.179080 21216 consensus_queue.cc:260] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } }
I20240502 01:55:55.184626 21216 ts_tablet_manager.cc:1398] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240502 01:55:55.185598 16511 tablet_copy_service.cc:342] P 39059854b64045698ded1ba4e8d47304: Request end of tablet copy session ed2e7e4188d0400b8689e28bbf5096ad-479fcadbde0449f4bfeef0bd2f82ebaf received from {username='slave'} at 127.31.161.197:49847
I20240502 01:55:55.185739 16511 tablet_copy_service.cc:433] P 39059854b64045698ded1ba4e8d47304: ending tablet copy session ed2e7e4188d0400b8689e28bbf5096ad-479fcadbde0449f4bfeef0bd2f82ebaf on tablet 479fcadbde0449f4bfeef0bd2f82ebaf with peer ed2e7e4188d0400b8689e28bbf5096ad
I20240502 01:55:55.194833 21145 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:55.199712 21288 tablet_copy_client.cc:783] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:55.200009 21288 tablet_copy_client.cc:647] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:55.200716 18645 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } } }
I20240502 01:55:55.202323 21288 tablet_copy_client.cc:523] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:55.208837 21299 ts_tablet_manager.cc:1888] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 01:55:55.209151 21299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18e676bcd8b04f18a6814d6ebbd1b488. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:55.209515 16882 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:55.210107 21043 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:55.220075 21288 tablet_bootstrap.cc:492] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:55.220265 21071 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, cdd4d50250184223992325c86cb6c68e (127.31.161.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: false } } }
I20240502 01:55:55.221750 18646 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, cdd4d50250184223992325c86cb6c68e (127.31.161.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: false } } }
I20240502 01:55:55.226881 16359 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, cdd4d50250184223992325c86cb6c68e (127.31.161.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: false } } }
I20240502 01:55:55.227843 16075 catalog_manager.cc:5484] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: config changed from index 11 to 12, cdd4d50250184223992325c86cb6c68e (127.31.161.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:55.229609 18913 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, cdd4d50250184223992325c86cb6c68e (127.31.161.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: false } } }
I20240502 01:55:55.231712 17016 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, cdd4d50250184223992325c86cb6c68e (127.31.161.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: false } } }
I20240502 01:55:55.236253 21064 raft_consensus.cc:1050] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304: attempting to promote NON_VOTER 703f6f043c6041a4b629933690580367 to VOTER
I20240502 01:55:55.237463 21064 consensus_queue.cc:237] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } }
I20240502 01:55:55.241039 16884 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, cdd4d50250184223992325c86cb6c68e (127.31.161.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: false } } }
I20240502 01:55:55.242766 18779 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:55.243968 19782 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:55.245492 21299 tablet_copy_client.cc:783] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:55.245795 21299 tablet_copy_client.cc:647] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:55.245973 17015 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:55.246165 16620 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [term 1 LEARNER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:55.246580 20578 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:55.246866 21062 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:55.248876 21299 tablet_copy_client.cc:523] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:55.250550 16754 consensus_queue.cc:237] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: false } }
I20240502 01:55:55.253758 18911 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:55.253813 16222 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:55.254792 20715 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:55.255241 20715 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:55.255940 16488 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:55.257673 16222 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, cdd4d50250184223992325c86cb6c68e (127.31.161.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: false } } }
I20240502 01:55:55.258438 16884 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:55.259047 21043 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:55.261387 21043 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:55.262059 18645 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:55.262095 18913 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:55.262749 20578 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:55.266031 21071 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:55.265754 16488 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, cdd4d50250184223992325c86cb6c68e (127.31.161.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } attrs { replace: true } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: false } } }
I20240502 01:55:55.269492 21299 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367: Bootstrap starting.
I20240502 01:55:55.278506 21043 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: false } } }
I20240502 01:55:55.279381 16884 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: false } } }
I20240502 01:55:55.283849 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 512eab02225d4544a342811f0a7359aa with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:55.285362 16488 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: false } } }
I20240502 01:55:55.291674 17014 raft_consensus.cc:1261] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:55.293208 21307 consensus_queue.cc:1035] T 512eab02225d4544a342811f0a7359aa P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:55.295063 16079 catalog_manager.cc:5484] T 512eab02225d4544a342811f0a7359aa P 6b9f9aaa1c86420190af75270914449a reported cstate change: config changed from index 12 to 13, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: false } } }
I20240502 01:55:55.294973 17014 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: false } } }
I20240502 01:55:55.299651 16339 tablet_service.cc:1514] Processing DeleteTablet for tablet 512eab02225d4544a342811f0a7359aa with delete_type TABLET_DATA_TOMBSTONED (TS dc9fcb8d12cb4ab28ec821c9473adeeb not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:43508
I20240502 01:55:55.299729 18645 raft_consensus.cc:1203] T bab1fc6db80c4e53a4f0cacccca74096 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 01:55:55.313252 16751 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:55.314477 16223 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:55.314584 21334 tablet_replica.cc:330] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb: stopping tablet replica
I20240502 01:55:55.314812 21334 raft_consensus.cc:2229] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:55.315270 21334 raft_consensus.cc:2258] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:55.315490 16882 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:55.315539 18911 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: false } } }
I20240502 01:55:55.316359 20578 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:55.316915 20578 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:55.317938 21334 ts_tablet_manager.cc:1875] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:55.320504 20578 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:55.325866 16224 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: false } } }
I20240502 01:55:55.327491 18645 raft_consensus.cc:2932] T 512eab02225d4544a342811f0a7359aa P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } attrs { promote: false } } }
I20240502 01:55:55.344478 20578 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:55.349700 18779 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:55.354218 16620 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:55.358774 16885 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:55.362012 21334 ts_tablet_manager.cc:1888] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240502 01:55:55.361658 16081 catalog_manager.cc:5484] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:55.362154 21334 log.cc:1196] T 512eab02225d4544a342811f0a7359aa P dc9fcb8d12cb4ab28ec821c9473adeeb: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-1/wal/wals/512eab02225d4544a342811f0a7359aa
I20240502 01:55:55.364143 16223 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:55.364902 21222 tablet_bootstrap.cc:492] T 3a4e2e002a7d441398f3da0ec650b8d2 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:55.365355 21222 tablet_bootstrap.cc:492] T 3a4e2e002a7d441398f3da0ec650b8d2 P aa7ec9abef1747e5a9e2404cee265c20: Bootstrap complete.
I20240502 01:55:55.365639 21222 ts_tablet_manager.cc:1367] T 3a4e2e002a7d441398f3da0ec650b8d2 P aa7ec9abef1747e5a9e2404cee265c20: Time spent bootstrapping tablet: real 0.444s user 0.047s sys 0.010s
I20240502 01:55:55.366412 21222 raft_consensus.cc:348] T 3a4e2e002a7d441398f3da0ec650b8d2 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:55.366732 21222 raft_consensus.cc:729] T 3a4e2e002a7d441398f3da0ec650b8d2 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: aa7ec9abef1747e5a9e2404cee265c20, State: Initialized, Role: LEARNER
I20240502 01:55:55.366947 21222 consensus_queue.cc:260] T 3a4e2e002a7d441398f3da0ec650b8d2 P aa7ec9abef1747e5a9e2404cee265c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: NON_VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: true } }
I20240502 01:55:55.368041 16070 catalog_manager.cc:4830] TS dc9fcb8d12cb4ab28ec821c9473adeeb (127.31.161.194:38133): tablet 512eab02225d4544a342811f0a7359aa (table rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]) successfully deleted
I20240502 01:55:55.370230 21216 ts_tablet_manager.cc:902] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad: Initiating tablet copy from peer 39059854b64045698ded1ba4e8d47304 (127.31.161.195:35693)
I20240502 01:55:55.379595 16751 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:55.396622 17015 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:55.398972 18779 raft_consensus.cc:1203] T 453dff7b204a431592ac15c058db0493 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240502 01:55:55.402890 18913 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:55.405442 16489 consensus_queue.cc:237] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } }
I20240502 01:55:55.409891 16620 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:55.413467 21062 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:55.421161 18778 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:55.421854 21325 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:55.427002 18913 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:55.427991 21325 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:55.426100 21222 ts_tablet_manager.cc:1398] T 3a4e2e002a7d441398f3da0ec650b8d2 P aa7ec9abef1747e5a9e2404cee265c20: Time spent starting tablet: real 0.060s user 0.002s sys 0.000s
I20240502 01:55:55.431038 18913 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:55.431866 16774 tablet_copy_service.cc:342] P ed2e7e4188d0400b8689e28bbf5096ad: Request end of tablet copy session aa7ec9abef1747e5a9e2404cee265c20-3a4e2e002a7d441398f3da0ec650b8d2 received from {username='slave'} at 127.31.161.202:42599
I20240502 01:55:55.431988 16774 tablet_copy_service.cc:433] P ed2e7e4188d0400b8689e28bbf5096ad: ending tablet copy session aa7ec9abef1747e5a9e2404cee265c20-3a4e2e002a7d441398f3da0ec650b8d2 on tablet 3a4e2e002a7d441398f3da0ec650b8d2 with peer aa7ec9abef1747e5a9e2404cee265c20
I20240502 01:55:55.432969 16754 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:55.433018 17015 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:55.433621 21348 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:55.435015 21325 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:55.440614 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1fa6224f34ef4bc58797f6cf801ff6da with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:55.440315 16079 catalog_manager.cc:5484] T 1fa6224f34ef4bc58797f6cf801ff6da P 0982e15b4c74425cad8d7d642eb63316 reported cstate change: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:55.443061 21218 tablet_bootstrap.cc:492] T 4da77826275d42b2b9e280c732575cfd P 0982e15b4c74425cad8d7d642eb63316: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:55.443475 21218 tablet_bootstrap.cc:492] T 4da77826275d42b2b9e280c732575cfd P 0982e15b4c74425cad8d7d642eb63316: Bootstrap complete.
I20240502 01:55:55.443775 21218 ts_tablet_manager.cc:1367] T 4da77826275d42b2b9e280c732575cfd P 0982e15b4c74425cad8d7d642eb63316: Time spent bootstrapping tablet: real 0.566s user 0.038s sys 0.013s
I20240502 01:55:55.444572 21218 raft_consensus.cc:348] T 4da77826275d42b2b9e280c732575cfd P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:55.447981 21062 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:55.449786 16864 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fa6224f34ef4bc58797f6cf801ff6da with delete_type TABLET_DATA_TOMBSTONED (TS 6b9f9aaa1c86420190af75270914449a not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:49752
I20240502 01:55:55.450013 16754 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:55.460222 16224 raft_consensus.cc:1261] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:55.460942 21064 consensus_queue.cc:1035] T 1fa6224f34ef4bc58797f6cf801ff6da P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:55.462502 16224 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:55.463683 16620 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:55.464216 18778 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:55.472959 21218 raft_consensus.cc:729] T 4da77826275d42b2b9e280c732575cfd P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0982e15b4c74425cad8d7d642eb63316, State: Initialized, Role: LEARNER
I20240502 01:55:55.473699 21216 tablet_copy_client.cc:308] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad: tablet copy: Beginning tablet copy session from remote peer at address 127.31.161.195:35693
I20240502 01:55:55.473379 21218 consensus_queue.cc:260] T 4da77826275d42b2b9e280c732575cfd P 0982e15b4c74425cad8d7d642eb63316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: NON_VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: true } }
I20240502 01:55:55.476918 16511 tablet_copy_service.cc:140] P 39059854b64045698ded1ba4e8d47304: Received BeginTabletCopySession request for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c from peer ed2e7e4188d0400b8689e28bbf5096ad ({username='slave'} at 127.31.161.197:49847)
I20240502 01:55:55.477115 16511 tablet_copy_service.cc:161] P 39059854b64045698ded1ba4e8d47304: Beginning new tablet copy session on tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c from peer ed2e7e4188d0400b8689e28bbf5096ad at {username='slave'} at 127.31.161.197:49847: session id = ed2e7e4188d0400b8689e28bbf5096ad-6150a46aa8ca4d9bbd16c141ad2f1a3c
I20240502 01:55:55.479949 16511 tablet_copy_source_session.cc:205] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 01:55:55.480933 21218 ts_tablet_manager.cc:1398] T 4da77826275d42b2b9e280c732575cfd P 0982e15b4c74425cad8d7d642eb63316: Time spent starting tablet: real 0.037s user 0.001s sys 0.004s
I20240502 01:55:55.485599 16905 tablet_copy_service.cc:342] P 6b9f9aaa1c86420190af75270914449a: Request end of tablet copy session 0982e15b4c74425cad8d7d642eb63316-4da77826275d42b2b9e280c732575cfd received from {username='slave'} at 127.31.161.200:50837
I20240502 01:55:55.485761 16905 tablet_copy_service.cc:433] P 6b9f9aaa1c86420190af75270914449a: ending tablet copy session 0982e15b4c74425cad8d7d642eb63316-4da77826275d42b2b9e280c732575cfd on tablet 4da77826275d42b2b9e280c732575cfd with peer 0982e15b4c74425cad8d7d642eb63316
I20240502 01:55:55.489691 21216 ts_tablet_manager.cc:1875] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 01:55:55.495960 21359 tablet_replica.cc:330] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a: stopping tablet replica
I20240502 01:55:55.496176 21359 raft_consensus.cc:2229] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:55.496670 21359 raft_consensus.cc:2258] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:55.500002 21359 ts_tablet_manager.cc:1875] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:55.505364 16754 raft_consensus.cc:1203] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 01:55:55.506366 17015 raft_consensus.cc:2932] T 1fa6224f34ef4bc58797f6cf801ff6da P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:55.512336 21216 ts_tablet_manager.cc:1888] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 01:55:55.512632 21216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 01:55:55.515220 21359 ts_tablet_manager.cc:1888] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240502 01:55:55.515779 21359 log.cc:1196] T 1fa6224f34ef4bc58797f6cf801ff6da P 6b9f9aaa1c86420190af75270914449a: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal/wals/1fa6224f34ef4bc58797f6cf801ff6da
I20240502 01:55:55.517179 16068 catalog_manager.cc:4830] TS 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121): tablet 1fa6224f34ef4bc58797f6cf801ff6da (table rebalance_test_table_4 [id=20ac001ea13f40de91661972e9e2d433]) successfully deleted
I20240502 01:55:55.536165 18778 raft_consensus.cc:1203] T 3a4e2e002a7d441398f3da0ec650b8d2 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 01:55:55.549198 21216 tablet_copy_client.cc:783] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad: tablet copy: Starting download of 0 data blocks...
I20240502 01:55:55.549464 21216 tablet_copy_client.cc:647] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad: tablet copy: Starting download of 1 WAL segments...
I20240502 01:55:55.551467 21216 tablet_copy_client.cc:523] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 01:55:55.559330 21288 tablet_bootstrap.cc:492] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:55.559870 21288 tablet_bootstrap.cc:492] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap complete.
I20240502 01:55:55.560173 21288 ts_tablet_manager.cc:1367] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.340s user 0.045s sys 0.004s
I20240502 01:55:55.563019 21216 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap starting.
I20240502 01:55:55.560977 21288 raft_consensus.cc:348] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } }
I20240502 01:55:55.569429 21288 raft_consensus.cc:729] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: LEARNER
I20240502 01:55:55.569669 21288 consensus_queue.cc:260] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } }
I20240502 01:55:55.570544 21198 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:55.570994 21198 tablet_bootstrap.cc:492] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: Bootstrap complete.
I20240502 01:55:55.571273 21198 ts_tablet_manager.cc:1367] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: Time spent bootstrapping tablet: real 0.565s user 0.045s sys 0.015s
I20240502 01:55:55.575805 21145 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:55.576205 21145 tablet_bootstrap.cc:492] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367: Bootstrap complete.
I20240502 01:55:55.576462 21145 ts_tablet_manager.cc:1367] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.382s user 0.054s sys 0.005s
I20240502 01:55:55.578423 21288 ts_tablet_manager.cc:1398] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20240502 01:55:55.581249 16642 tablet_copy_service.cc:342] P 703f6f043c6041a4b629933690580367: Request end of tablet copy session ed2e7e4188d0400b8689e28bbf5096ad-f53b466fba6a4655a63a559b0ea56f77 received from {username='slave'} at 127.31.161.197:37875
I20240502 01:55:55.581354 16642 tablet_copy_service.cc:433] P 703f6f043c6041a4b629933690580367: ending tablet copy session ed2e7e4188d0400b8689e28bbf5096ad-f53b466fba6a4655a63a559b0ea56f77 on tablet f53b466fba6a4655a63a559b0ea56f77 with peer ed2e7e4188d0400b8689e28bbf5096ad
I20240502 01:55:55.572067 21198 raft_consensus.cc:348] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: true } }
I20240502 01:55:55.585403 21198 raft_consensus.cc:729] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 6b9f9aaa1c86420190af75270914449a, State: Initialized, Role: LEARNER
I20240502 01:55:55.585646 21198 consensus_queue.cc:260] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: NON_VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: true } }
I20240502 01:55:55.586346 21198 ts_tablet_manager.cc:1398] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 01:55:55.588354 18667 tablet_copy_service.cc:342] P a7c4573db86a415285d274ca57dde659: Request end of tablet copy session 6b9f9aaa1c86420190af75270914449a-48a81a0b7d47479e929d80041f34a3b1 received from {username='slave'} at 127.31.161.198:55601
I20240502 01:55:55.588541 18667 tablet_copy_service.cc:433] P a7c4573db86a415285d274ca57dde659: ending tablet copy session 6b9f9aaa1c86420190af75270914449a-48a81a0b7d47479e929d80041f34a3b1 on tablet 48a81a0b7d47479e929d80041f34a3b1 with peer 6b9f9aaa1c86420190af75270914449a
I20240502 01:55:55.592949 21145 raft_consensus.cc:348] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } }
I20240502 01:55:55.593376 21145 raft_consensus.cc:729] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: LEARNER
I20240502 01:55:55.593930 21145 consensus_queue.cc:260] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } }
I20240502 01:55:55.595942 21145 ts_tablet_manager.cc:1398] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.019s user 0.004s sys 0.000s
I20240502 01:55:55.596974 17036 tablet_copy_service.cc:342] P a6575ebf978a4631a13ba5c243c3c4f9: Request end of tablet copy session 703f6f043c6041a4b629933690580367-a8f7e3c8b6d6434db2a99d7139599a5d received from {username='slave'} at 127.31.161.196:37261
I20240502 01:55:55.597149 17036 tablet_copy_service.cc:433] P a6575ebf978a4631a13ba5c243c3c4f9: ending tablet copy session 703f6f043c6041a4b629933690580367-a8f7e3c8b6d6434db2a99d7139599a5d on tablet a8f7e3c8b6d6434db2a99d7139599a5d with peer 703f6f043c6041a4b629933690580367
I20240502 01:55:55.606680 21299 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:55.607213 21299 tablet_bootstrap.cc:492] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367: Bootstrap complete.
I20240502 01:55:55.607542 21299 ts_tablet_manager.cc:1367] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367: Time spent bootstrapping tablet: real 0.338s user 0.051s sys 0.004s
I20240502 01:55:55.608325 21299 raft_consensus.cc:348] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } }
I20240502 01:55:55.609035 21299 raft_consensus.cc:729] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 703f6f043c6041a4b629933690580367, State: Initialized, Role: LEARNER
I20240502 01:55:55.609333 21299 consensus_queue.cc:260] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: NON_VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: true } }
I20240502 01:55:55.614187 21299 ts_tablet_manager.cc:1398] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 01:55:55.616276 16511 tablet_copy_service.cc:342] P 39059854b64045698ded1ba4e8d47304: Request end of tablet copy session 703f6f043c6041a4b629933690580367-18e676bcd8b04f18a6814d6ebbd1b488 received from {username='slave'} at 127.31.161.196:58449
I20240502 01:55:55.616396 16511 tablet_copy_service.cc:433] P 39059854b64045698ded1ba4e8d47304: ending tablet copy session 703f6f043c6041a4b629933690580367-18e676bcd8b04f18a6814d6ebbd1b488 on tablet 18e676bcd8b04f18a6814d6ebbd1b488 with peer 703f6f043c6041a4b629933690580367
I20240502 01:55:55.643373 18913 raft_consensus.cc:1203] T 4da77826275d42b2b9e280c732575cfd P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 01:55:55.678587 21216 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 01:55:55.679167 21216 tablet_bootstrap.cc:492] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad: Bootstrap complete.
I20240502 01:55:55.679503 21216 ts_tablet_manager.cc:1367] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad: Time spent bootstrapping tablet: real 0.117s user 0.046s sys 0.008s
I20240502 01:55:55.680337 21216 raft_consensus.cc:348] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } }
I20240502 01:55:55.680791 21216 raft_consensus.cc:729] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ed2e7e4188d0400b8689e28bbf5096ad, State: Initialized, Role: LEARNER
I20240502 01:55:55.681098 21216 consensus_queue.cc:260] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: NON_VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: true } }
I20240502 01:55:55.681803 21216 ts_tablet_manager.cc:1398] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 01:55:55.683775 16511 tablet_copy_service.cc:342] P 39059854b64045698ded1ba4e8d47304: Request end of tablet copy session ed2e7e4188d0400b8689e28bbf5096ad-6150a46aa8ca4d9bbd16c141ad2f1a3c received from {username='slave'} at 127.31.161.197:49847
I20240502 01:55:55.683912 16511 tablet_copy_service.cc:433] P 39059854b64045698ded1ba4e8d47304: ending tablet copy session ed2e7e4188d0400b8689e28bbf5096ad-6150a46aa8ca4d9bbd16c141ad2f1a3c on tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c with peer ed2e7e4188d0400b8689e28bbf5096ad
I20240502 01:55:55.693480 16882 raft_consensus.cc:1203] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20240502 01:55:55.750764 16754 raft_consensus.cc:1203] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240502 01:55:55.830554 20911 raft_consensus.cc:1050] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9: attempting to promote NON_VOTER a7c4573db86a415285d274ca57dde659 to VOTER
I20240502 01:55:55.831625 20911 consensus_queue.cc:237] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:55.832316 21285 raft_consensus.cc:1050] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a: attempting to promote NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 to VOTER
I20240502 01:55:55.833527 21285 consensus_queue.cc:237] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:55.836297 18645 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P a7c4573db86a415285d274ca57dde659 [term 1 LEARNER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:55.836418 16489 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:55.837046 20913 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:55.837306 18779 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:55.838405 21050 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:55.839015 21050 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:55.849589 16620 raft_consensus.cc:1203] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240502 01:55:55.853976 18779 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:55.854663 21016 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:55.859081 16224 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:55.859398 18646 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:55.860385 16882 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:55.860488 16754 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:55.860489 21050 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:55.860971 21284 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:55.861543 21050 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:55.863502 16359 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:55.863792 21050 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:55.864089 20913 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:55.866801 16751 raft_consensus.cc:1203] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 01:55:55.876116 18913 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:55.876462 16489 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:55.877113 21016 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:55.877662 21016 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:55.887391 16358 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:55.887786 17015 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:55.888237 21284 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:55.889530 21016 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:55.892990 21284 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:55.898675 21435 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:55.898725 17016 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:55.900005 16359 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:55.902060 16223 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:55.906764 18778 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:55.907126 16489 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:55.915406 16080 catalog_manager.cc:5484] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:55.921651 18913 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:55.927836 16081 catalog_manager.cc:5484] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316 reported cstate change: config changed from index 11 to 12, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6b9f9aaa1c86420190af75270914449a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:55.929102 16620 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:55.930509 16751 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:55.932955 21016 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:55.938387 16620 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:55.941058 18779 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:55.944396 16489 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:55.947012 18646 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:55.958714 16358 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:55.980479 16884 consensus_queue.cc:237] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:55.983934 17016 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:55.985433 16489 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:55.985483 21285 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:55.986610 21455 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:55.986852 18913 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:55.988615 21285 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:55.988749 16358 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:55.989414 21285 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:55.990578 18779 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:55.991055 21016 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:55.994180 18779 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:55.998539 17016 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:55.998735 16488 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:56.005101 16076 catalog_manager.cc:5484] T 453dff7b204a431592ac15c058db0493 P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: config changed from index 12 to 13, VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) evicted. New cstate: current_term: 1 leader_uuid: "6b9f9aaa1c86420190af75270914449a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:56.010697 18626 tablet_service.cc:1514] Processing DeleteTablet for tablet 453dff7b204a431592ac15c058db0493 with delete_type TABLET_DATA_TOMBSTONED (TS a7c4573db86a415285d274ca57dde659 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:33074
I20240502 01:55:56.011706 21461 tablet_replica.cc:330] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659: stopping tablet replica
I20240502 01:55:56.012471 21461 raft_consensus.cc:2229] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:56.013134 21461 raft_consensus.cc:2258] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:55.977131 18646 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:56.014420 21461 ts_tablet_manager.cc:1875] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:56.016386 16885 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a7c4573db86a415285d274ca57dde659 (127.31.161.201) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false replace: true } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:56.023129 21461 ts_tablet_manager.cc:1888] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240502 01:55:56.023341 21461 log.cc:1196] T 453dff7b204a431592ac15c058db0493 P a7c4573db86a415285d274ca57dde659: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/wal/wals/453dff7b204a431592ac15c058db0493
I20240502 01:55:56.024427 16069 catalog_manager.cc:4830] TS a7c4573db86a415285d274ca57dde659 (127.31.161.201:44737): tablet 453dff7b204a431592ac15c058db0493 (table rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]) successfully deleted
I20240502 01:55:56.028318 17016 consensus_queue.cc:237] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } }
I20240502 01:55:56.029884 21285 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:56.032021 16359 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.033030 16488 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.033375 21440 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:56.033886 16223 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.034667 16754 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.034651 20911 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:56.035187 20911 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:56.035651 20911 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:56.037233 16068 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 453dff7b204a431592ac15c058db0493 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:56.039227 16882 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.041924 21435 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:56.042765 21435 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:56.044370 21455 raft_consensus.cc:1050] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a: attempting to promote NON_VOTER 0982e15b4c74425cad8d7d642eb63316 to VOTER
I20240502 01:55:56.045513 21455 consensus_queue.cc:237] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:56.049373 16621 raft_consensus.cc:1203] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240502 01:55:56.051342 16223 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:56.053544 16488 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:56.058409 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bab1fc6db80c4e53a4f0cacccca74096 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:56.065796 21325 raft_consensus.cc:1050] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304: attempting to promote NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad to VOTER
I20240502 01:55:56.065755 16076 catalog_manager.cc:5484] T bab1fc6db80c4e53a4f0cacccca74096 P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index 9 to 10, VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) evicted. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:56.066936 21325 consensus_queue.cc:237] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } }
I20240502 01:55:56.074447 17015 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.075039 18913 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P 0982e15b4c74425cad8d7d642eb63316 [term 1 LEARNER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.075235 16754 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.076857 18647 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.083981 21016 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.008s
I20240502 01:55:56.091519 21016 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.092123 21016 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.092780 16488 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.093451 21016 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.101351 16882 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:56.111284 16359 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:56.112280 16751 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:56.122023 21284 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.127497 16359 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.127725 16223 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.128392 21455 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.128945 21457 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.134593 18645 raft_consensus.cc:1261] T bab1fc6db80c4e53a4f0cacccca74096 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.137636 16751 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEARNER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.138159 19782 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.141137 17015 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.142266 19782 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.143905 21309 raft_consensus.cc:1050] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad: attempting to promote NON_VOTER aa7ec9abef1747e5a9e2404cee265c20 to VOTER
I20240502 01:55:56.145380 21309 consensus_queue.cc:237] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:56.147925 16885 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.148540 19782 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.149375 18760 tablet_service.cc:1514] Processing DeleteTablet for tablet bab1fc6db80c4e53a4f0cacccca74096 with delete_type TABLET_DATA_TOMBSTONED (TS aa7ec9abef1747e5a9e2404cee265c20 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:46028
I20240502 01:55:56.149909 16621 raft_consensus.cc:1261] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:56.150264 18911 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:56.150743 21016 consensus_queue.cc:1035] T 453dff7b204a431592ac15c058db0493 P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:56.150902 16358 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:56.152202 16621 raft_consensus.cc:2932] T 453dff7b204a431592ac15c058db0493 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:56.158437 16620 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.159013 19782 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.164517 21489 tablet_replica.cc:330] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20: stopping tablet replica
I20240502 01:55:56.165064 21489 raft_consensus.cc:2229] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:56.165464 21489 raft_consensus.cc:2258] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:56.166008 20913 consensus_queue.cc:1035] T bab1fc6db80c4e53a4f0cacccca74096 P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:56.167178 21489 ts_tablet_manager.cc:1875] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 01:55:56.167223 21469 log.cc:924] Time spent T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20240502 01:55:56.168280 21335 raft_consensus.cc:1050] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659: attempting to promote NON_VOTER 6b9f9aaa1c86420190af75270914449a to VOTER
I20240502 01:55:56.168102 18645 raft_consensus.cc:2932] T bab1fc6db80c4e53a4f0cacccca74096 P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } }
I20240502 01:55:56.169705 21335 consensus_queue.cc:237] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 9, Current term: 2, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: false } }
I20240502 01:55:56.174980 18646 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.175523 19782 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.187151 16358 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.187819 21064 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.189734 21489 ts_tablet_manager.cc:1888] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 01:55:56.190393 21489 log.cc:1196] T bab1fc6db80c4e53a4f0cacccca74096 P aa7ec9abef1747e5a9e2404cee265c20: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-9/wal/wals/bab1fc6db80c4e53a4f0cacccca74096
I20240502 01:55:56.192191 16068 catalog_manager.cc:4830] TS aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202:43977): tablet bab1fc6db80c4e53a4f0cacccca74096 (table rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]) successfully deleted
I20240502 01:55:56.195947 16357 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 01:55:56.196462 18911 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316 [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 01:55:56.196527 16224 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 01:55:56.197115 21337 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:56.197794 21337 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:56.198287 21337 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:56.202819 16884 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [term 2 LEARNER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 01:55:56.206375 20887 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:56.214236 16620 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 01:55:56.215718 20887 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:56.219946 17015 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 01:55:56.220577 20887 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:56.226794 20913 raft_consensus.cc:1050] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9: attempting to promote NON_VOTER 703f6f043c6041a4b629933690580367 to VOTER
I20240502 01:55:56.227957 20913 consensus_queue.cc:237] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } }
I20240502 01:55:56.228341 16224 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.228928 19782 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.230867 16752 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 01:55:56.231570 20887 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:56.252982 21457 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:56.254220 16222 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:56.259045 17016 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:56.262218 16076 catalog_manager.cc:5484] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e reported cstate change: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6b9f9aaa1c86420190af75270914449a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:56.262207 16488 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
W20240502 01:55:56.263429 18581 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.193:41953, user_credentials={real_user=slave}} blocked reactor thread for 66394.2us
I20240502 01:55:56.271481 18913 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:56.281579 21325 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.275381 18647 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:56.286046 16356 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.292379 16752 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.294319 17015 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.295218 18646 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.299535 16075 catalog_manager.cc:5484] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index 8 to 9, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.300668 16885 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.304646 16359 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:56.307343 19782 raft_consensus.cc:1050] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304: attempting to promote NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad to VOTER
I20240502 01:55:56.307431 16754 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:56.313064 19782 consensus_queue.cc:237] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } }
I20240502 01:55:56.314366 16225 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.318404 18912 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:56.319160 16883 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:56.319729 20911 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:56.320349 20911 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:56.320807 18644 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:56.321352 20911 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:56.331319 16752 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEARNER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:56.331987 21064 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:56.334236 16751 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:56.334861 21468 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:56.337435 16223 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:56.339737 21468 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.002s
I20240502 01:55:56.340797 16619 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [term 1 LEARNER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:56.342193 20913 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:56.349535 18779 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:56.350160 21064 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:56.354723 18778 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 LEARNER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.356040 16489 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.356571 21043 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.357336 21043 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.366919 18911 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.368680 21473 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.337116 16357 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:56.373522 16602 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:56.375174 16357 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.375870 16224 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.376060 21473 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.376677 21043 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.378121 17016 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.379815 16620 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:56.380450 21043 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.380553 21064 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:56.384971 21468 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:56.389067 16621 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
W20240502 01:55:56.403250 18582 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:38133, user_credentials={real_user=slave}} blocked reactor thread for 91876.9us
W20240502 01:55:56.404026 18580 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.200:38559, user_credentials={real_user=slave}} blocked reactor thread for 62144.6us
I20240502 01:55:56.404418 21497 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 6b9f9aaa1c86420190af75270914449a (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: false } } }
I20240502 01:55:56.405670 18910 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 6b9f9aaa1c86420190af75270914449a (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: false } } }
I20240502 01:55:56.410045 18645 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:56.413852 16356 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 6b9f9aaa1c86420190af75270914449a (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: false } } }
I20240502 01:55:56.415467 21514 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:56.421183 16080 catalog_manager.cc:5484] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb reported cstate change: config changed from index 12 to 13, 6b9f9aaa1c86420190af75270914449a (127.31.161.198) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a7c4573db86a415285d274ca57dde659" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: false } } }
I20240502 01:55:56.426854 16221 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 6b9f9aaa1c86420190af75270914449a (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: false } } }
I20240502 01:55:56.428315 16471 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:56.441886 16486 consensus_queue.cc:237] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } }
I20240502 01:55:56.442760 18647 consensus_queue.cc:237] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 9, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: false } }
I20240502 01:55:56.446794 16751 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.447010 16356 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.449231 19782 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:56.449507 16359 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:56.450106 19782 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 01:55:56.453805 16821 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.194:38133, user_credentials={real_user=slave}} blocked reactor thread for 91350.4us
W20240502 01:55:56.455111 16818 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.199:46837, user_credentials={real_user=slave}} blocked reactor thread for 88338.9us
I20240502 01:55:56.456269 16881 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.456889 21043 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.456519 16884 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 6b9f9aaa1c86420190af75270914449a (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: false } } }
I20240502 01:55:56.455906 18913 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316 [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 01:55:56.458169 21497 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 01:55:56.461738 16356 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 01:55:56.462287 21497 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 01:55:56.465382 16884 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 01:55:56.466238 17014 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 01:55:56.466940 21335 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:56.467448 21335 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 01:55:56.468354 21064 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:56.474452 18760 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:56.475545 17014 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 6b9f9aaa1c86420190af75270914449a (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: false } } }
I20240502 01:55:56.475917 16754 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 6b9f9aaa1c86420190af75270914449a (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: false } } }
I20240502 01:55:56.481422 16883 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.482029 21325 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:56.487630 17013 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:56.488066 17012 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.488240 21064 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:56.495266 21064 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:56.495409 16221 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 01:55:56.497968 18893 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:56.498802 16885 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:56.501814 21209 raft_consensus.cc:1050] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367: attempting to promote NON_VOTER ed2e7e4188d0400b8689e28bbf5096ad to VOTER
I20240502 01:55:56.501493 21335 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: false } } }
I20240502 01:55:56.503017 21209 consensus_queue.cc:237] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } }
I20240502 01:55:56.504022 21514 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:56.507323 16069 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 48a81a0b7d47479e929d80041f34a3b1 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:56.508594 18913 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: false } } }
I20240502 01:55:56.512661 21043 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:56.515033 16075 catalog_manager.cc:5484] T 48a81a0b7d47479e929d80041f34a3b1 P 0982e15b4c74425cad8d7d642eb63316 reported cstate change: config changed from index 13 to 14, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New cstate: current_term: 2 leader_uuid: "a7c4573db86a415285d274ca57dde659" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: false } } }
I20240502 01:55:56.515836 21497 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 01:55:56.520803 16733 tablet_service.cc:1514] Processing DeleteTablet for tablet 48a81a0b7d47479e929d80041f34a3b1 with delete_type TABLET_DATA_TOMBSTONED (TS ed2e7e4188d0400b8689e28bbf5096ad not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:47194
I20240502 01:55:56.519275 16884 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 6b9f9aaa1c86420190af75270914449a [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: false } } }
I20240502 01:55:56.524003 16221 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P cdd4d50250184223992325c86cb6c68e [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: false } } }
I20240502 01:55:56.529790 16489 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:56.530615 21064 raft_consensus.cc:1050] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304: attempting to promote NON_VOTER 703f6f043c6041a4b629933690580367 to VOTER
I20240502 01:55:56.532601 20911 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:56.534446 16222 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:56.536283 21325 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.537765 16880 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:56.539726 16075 catalog_manager.cc:5484] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad reported cstate change: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:56.546012 21064 consensus_queue.cc:237] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } }
I20240502 01:55:56.550895 18647 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:56.553051 18645 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.552883 16080 catalog_manager.cc:5484] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 01:55:56.556177 17014 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:56.556280 16357 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:56.567812 19782 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20240502 01:55:56.566475 16355 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: false } } }
I20240502 01:55:56.560933 16358 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:56.575898 18647 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
W20240502 01:55:56.577852 21515 log.cc:924] Time spent T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316: Append to log took a long time: real 0.129s user 0.000s sys 0.000s
I20240502 01:55:56.583526 16752 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:56.583896 16075 catalog_manager.cc:5484] T 479fcadbde0449f4bfeef0bd2f82ebaf P a7c4573db86a415285d274ca57dde659 reported cstate change: config changed from index 9 to 10, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.586159 18912 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:56.578477 16883 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.576175 17015 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P a6575ebf978a4631a13ba5c243c3c4f9 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: false } } }
I20240502 01:55:56.593735 18626 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 01:55:56.596199 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 479fcadbde0449f4bfeef0bd2f82ebaf with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:56.602047 17012 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.624392 21514 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.622181 18780 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.627386 16602 tablet_service.cc:1514] Processing DeleteTablet for tablet 479fcadbde0449f4bfeef0bd2f82ebaf with delete_type TABLET_DATA_TOMBSTONED (TS 703f6f043c6041a4b629933690580367 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:53048
I20240502 01:55:56.528859 16622 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 6b9f9aaa1c86420190af75270914449a (127.31.161.198) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: false } } }
W20240502 01:55:56.634073 21522 log.cc:924] Time spent T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20240502 01:55:56.637141 16621 raft_consensus.cc:1261] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 2 FOLLOWER]: Refusing update from remote peer a7c4573db86a415285d274ca57dde659: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 01:55:56.596683 18778 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:56.638367 21337 consensus_queue.cc:1035] T 48a81a0b7d47479e929d80041f34a3b1 P a7c4573db86a415285d274ca57dde659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 01:55:56.640388 16622 raft_consensus.cc:2932] T 48a81a0b7d47479e929d80041f34a3b1 P 703f6f043c6041a4b629933690580367 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { promote: false } } }
I20240502 01:55:56.641790 21555 tablet_replica.cc:330] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367: stopping tablet replica
I20240502 01:55:56.642063 21555 raft_consensus.cc:2229] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:56.642630 21555 raft_consensus.cc:2258] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 01:55:56.643479 16556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.198:38121, user_credentials={real_user=slave}} blocked reactor thread for 111817us
I20240502 01:55:56.649053 18912 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:56.649964 19782 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 01:55:56.652294 21532 log.cc:924] Time spent T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367: Append to log took a long time: real 0.166s user 0.000s sys 0.000s
I20240502 01:55:56.658324 18779 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:56.635008 16885 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.622666 16884 consensus_queue.cc:237] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } }
I20240502 01:55:56.671305 18909 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.670657 18913 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:56.675601 21285 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:56.677161 21546 tablet_replica.cc:330] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad: stopping tablet replica
I20240502 01:55:56.677421 21546 raft_consensus.cc:2229] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:56.678403 21546 raft_consensus.cc:2258] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:56.681751 21555 ts_tablet_manager.cc:1875] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:56.682097 19782 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:56.686584 17012 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.687250 21488 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:56.683897 16752 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.690398 16751 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.691671 21488 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:56.694276 21488 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:56.696713 16754 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:56.699385 16068 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4da77826275d42b2b9e280c732575cfd with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:56.702590 18647 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.704010 16489 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.704689 16620 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:56.706714 16357 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.708480 16358 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.710469 16075 catalog_manager.cc:5484] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659 reported cstate change: config changed from index 11 to 12, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.710258 16359 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.716997 21546 ts_tablet_manager.cc:1875] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:56.717324 16221 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.717939 20634 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.720629 16617 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.721645 18777 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.723826 21488 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:56.724305 21285 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:56.728823 17012 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:56.725419 18913 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:56.732645 16751 consensus_queue.cc:237] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } }
I20240502 01:55:56.735426 16357 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.736230 16225 raft_consensus.cc:1261] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 6b9f9aaa1c86420190af75270914449a: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.735996 16884 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, aa7ec9abef1747e5a9e2404cee265c20 (127.31.161.202) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:56.737296 16357 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.737771 16864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 01:55:56.737938 21043 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20240502 01:55:56.741500 16080 catalog_manager.cc:5484] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a reported cstate change: config changed from index 9 to 10, VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) evicted. New cstate: current_term: 1 leader_uuid: "6b9f9aaa1c86420190af75270914449a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 01:55:56.743948 16995 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 01:55:56.746220 17015 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.746670 18626 tablet_service.cc:1514] Processing DeleteTablet for tablet 4da77826275d42b2b9e280c732575cfd with delete_type TABLET_DATA_TOMBSTONED (TS a7c4573db86a415285d274ca57dde659 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:33074
I20240502 01:55:56.746850 21529 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:56.748210 20634 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.748814 20634 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.738716 16224 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.749938 21043 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:56.752735 21457 consensus_queue.cc:1035] T 4da77826275d42b2b9e280c732575cfd P 6b9f9aaa1c86420190af75270914449a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:56.752414 21529 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:56.754568 16223 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:56.755314 16224 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:56.758036 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3a4e2e002a7d441398f3da0ec650b8d2 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:56.758534 16220 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:56.759116 16219 raft_consensus.cc:1261] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.760268 18777 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.760943 21529 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:56.761615 17014 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:56.762435 18777 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:56.764933 16079 catalog_manager.cc:5484] T 3a4e2e002a7d441398f3da0ec650b8d2 P cdd4d50250184223992325c86cb6c68e reported cstate change: config changed from index 9 to 10, VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) evicted. New cstate: current_term: 1 leader_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:56.784165 16751 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 LEARNER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.785918 17014 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.787676 16882 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.789008 16752 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
W20240502 01:55:56.791141 16424 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.198:38121, user_credentials={real_user=slave}} blocked reactor thread for 57716us
W20240502 01:55:56.791783 16426 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.196:40425, user_credentials={real_user=slave}} blocked reactor thread for 55806.6us
I20240502 01:55:56.792009 20634 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.792732 20634 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.800575 20634 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.802063 21558 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:56.802248 21568 consensus_queue.cc:1035] T 479fcadbde0449f4bfeef0bd2f82ebaf P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:56.803261 17012 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:56.803788 21567 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:56.804785 16489 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:56.807377 21555 ts_tablet_manager.cc:1888] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 01:55:56.812227 16488 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.812366 16880 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:56.813383 16753 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:56.813246 16221 raft_consensus.cc:2932] T 479fcadbde0449f4bfeef0bd2f82ebaf P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 703f6f043c6041a4b629933690580367 (127.31.161.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.814100 16486 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 01:55:56.815012 21567 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:56.815846 21541 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 01:55:56.820307 16224 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:56.832126 16357 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:56.839605 21571 tablet_replica.cc:330] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659: stopping tablet replica
I20240502 01:55:56.839886 21571 raft_consensus.cc:2229] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:56.840296 21571 raft_consensus.cc:2258] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:56.839784 17016 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.842756 16358 raft_consensus.cc:2932] T 4da77826275d42b2b9e280c732575cfd P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a7c4573db86a415285d274ca57dde659 (127.31.161.201) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } }
I20240502 01:55:56.844602 21568 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:56.849614 17015 consensus_queue.cc:237] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } }
I20240502 01:55:56.849380 16881 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:56.858772 17012 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:56.862469 21571 ts_tablet_manager.cc:1875] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:56.815683 16620 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [term 1 LEARNER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 01:55:56.840751 21569 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 01:55:56.870558 21555 log.cc:1196] T 479fcadbde0449f4bfeef0bd2f82ebaf P 703f6f043c6041a4b629933690580367: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-3/wal/wals/479fcadbde0449f4bfeef0bd2f82ebaf
I20240502 01:55:56.872007 16070 catalog_manager.cc:4830] TS 703f6f043c6041a4b629933690580367 (127.31.161.196:40425): tablet 479fcadbde0449f4bfeef0bd2f82ebaf (table rebalance_test_table_6 [id=f6db8b099ff54fe5968ce1b26165cf74]) successfully deleted
I20240502 01:55:56.872453 21541 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20240502 01:55:56.872551 16885 raft_consensus.cc:1261] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer ed2e7e4188d0400b8689e28bbf5096ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:56.883672 16486 consensus_queue.cc:237] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } }
I20240502 01:55:56.885906 21550 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.886996 16204 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:56.888478 16357 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.888731 16488 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.896804 16224 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.901625 16885 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.902987 18646 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:56.903689 21514 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:56.904829 18777 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:56.905479 21558 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:56.906934 18780 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.907721 16079 catalog_manager.cc:5484] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a reported cstate change: config changed from index 8 to 9, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.909938 16617 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:56.911513 21541 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:56.915191 16339 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 01:55:56.915371 17014 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:56.915763 16358 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:56.915979 21541 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:56.917390 21578 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:56.917225 21514 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.919159 16357 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.881559 16620 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:56.914899 21571 ts_tablet_manager.cc:1888] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 01:55:56.922430 21571 log.cc:1196] T 4da77826275d42b2b9e280c732575cfd P a7c4573db86a415285d274ca57dde659: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-8/wal/wals/4da77826275d42b2b9e280c732575cfd
I20240502 01:55:56.923558 16617 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.926568 16069 catalog_manager.cc:4830] TS a7c4573db86a415285d274ca57dde659 (127.31.161.201:44737): tablet 4da77826275d42b2b9e280c732575cfd (table rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]) successfully deleted
I20240502 01:55:56.927960 21529 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.065s
I20240502 01:55:56.938931 16753 raft_consensus.cc:1261] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:56.943121 16221 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:56.946099 18913 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:56.956231 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:56.965628 18893 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a4e2e002a7d441398f3da0ec650b8d2 with delete_type TABLET_DATA_TOMBSTONED (TS 0982e15b4c74425cad8d7d642eb63316 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:58664
I20240502 01:55:56.972221 21541 consensus_queue.cc:1035] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:56.972327 16075 catalog_manager.cc:5484] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 reported cstate change: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:56.973327 16075 catalog_manager.cc:5484] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 703f6f043c6041a4b629933690580367 reported cstate change: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.979144 18646 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.979414 17014 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.980196 18778 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:56.984719 18777 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:56.994338 16864 tablet_service.cc:1514] Processing DeleteTablet for tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c with delete_type TABLET_DATA_TOMBSTONED (TS 6b9f9aaa1c86420190af75270914449a not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:49752
I20240502 01:55:56.938220 16754 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 703f6f043c6041a4b629933690580367 (127.31.161.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { replace: true } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:56.974476 16753 raft_consensus.cc:2932] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:57.009394 21529 consensus_queue.cc:1035] T 3a4e2e002a7d441398f3da0ec650b8d2 P ed2e7e4188d0400b8689e28bbf5096ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20240502 01:55:57.014879 16489 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:57.025591 21600 tablet_replica.cc:330] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316: stopping tablet replica
I20240502 01:55:57.034046 16617 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:57.034752 20911 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:57.034404 16754 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:57.034780 16622 consensus_queue.cc:237] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } }
I20240502 01:55:57.038885 16225 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:57.039481 20634 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:57.040800 18780 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:57.041325 17013 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } attrs { replace: true } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:57.047957 20634 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:57.048187 17016 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:57.048355 16489 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:57.049645 18913 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:57.050170 21569 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:57.050285 20911 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:57.050612 21554 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:57.052223 16751 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 01:55:57.052889 20634 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:57.055279 21546 ts_tablet_manager.cc:1888] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240502 01:55:57.055447 21546 log.cc:1196] T 48a81a0b7d47479e929d80041f34a3b1 P ed2e7e4188d0400b8689e28bbf5096ad: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal/wals/48a81a0b7d47479e929d80041f34a3b1
I20240502 01:55:57.056411 16067 catalog_manager.cc:4830] TS ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825): tablet 48a81a0b7d47479e929d80041f34a3b1 (table rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]) successfully deleted
I20240502 01:55:57.056525 20634 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:57.059083 16733 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 01:55:57.059182 16486 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:57.062011 16070 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f53b466fba6a4655a63a559b0ea56f77 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:57.062997 16753 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:57.066615 16358 raft_consensus.cc:1261] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer 703f6f043c6041a4b629933690580367: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 01:55:57.064167 16688 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.161.202:43977, user_credentials={real_user=slave}} blocked reactor thread for 186524us
I20240502 01:55:57.070017 16881 raft_consensus.cc:2932] T 3a4e2e002a7d441398f3da0ec650b8d2 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } }
I20240502 01:55:57.073498 21569 consensus_queue.cc:1035] T f53b466fba6a4655a63a559b0ea56f77 P 703f6f043c6041a4b629933690580367 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 01:55:57.073343 16079 catalog_manager.cc:5484] T f53b466fba6a4655a63a559b0ea56f77 P 39059854b64045698ded1ba4e8d47304 reported cstate change: config changed from index 9 to 10, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New cstate: current_term: 1 leader_uuid: "703f6f043c6041a4b629933690580367" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:57.077663 16863 tablet_service.cc:1514] Processing DeleteTablet for tablet f53b466fba6a4655a63a559b0ea56f77 with delete_type TABLET_DATA_TOMBSTONED (TS 6b9f9aaa1c86420190af75270914449a not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:49752
I20240502 01:55:57.076889 17013 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:57.077479 16358 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:57.084085 16357 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:57.084429 16225 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:57.084767 20911 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:57.085182 20913 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:57.087390 21440 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:57.087795 16357 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P dc9fcb8d12cb4ab28ec821c9473adeeb [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:57.091796 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a8f7e3c8b6d6434db2a99d7139599a5d with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:57.092950 18646 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:57.093808 20913 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:57.094732 16754 raft_consensus.cc:1261] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Refusing update from remote peer a6575ebf978a4631a13ba5c243c3c4f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:57.095880 21601 tablet_replica.cc:330] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a: stopping tablet replica
I20240502 01:55:57.096097 21601 raft_consensus.cc:2229] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:57.095613 18647 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:57.096386 21601 raft_consensus.cc:2258] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:57.096448 20913 consensus_queue.cc:1035] T a8f7e3c8b6d6434db2a99d7139599a5d P a6575ebf978a4631a13ba5c243c3c4f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:57.097556 16221 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:57.102864 16079 catalog_manager.cc:5484] T a8f7e3c8b6d6434db2a99d7139599a5d P a7c4573db86a415285d274ca57dde659 reported cstate change: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New cstate: current_term: 1 leader_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:57.107692 16864 tablet_service.cc:1514] Processing DeleteTablet for tablet a8f7e3c8b6d6434db2a99d7139599a5d with delete_type TABLET_DATA_TOMBSTONED (TS 6b9f9aaa1c86420190af75270914449a not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:49752
I20240502 01:55:57.110018 16754 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:57.122514 18913 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:57.128175 16620 raft_consensus.cc:2932] T a8f7e3c8b6d6434db2a99d7139599a5d P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:57.133189 18780 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:57.143733 16225 raft_consensus.cc:2932] T f53b466fba6a4655a63a559b0ea56f77 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 6b9f9aaa1c86420190af75270914449a (127.31.161.198) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } attrs { promote: false } } }
I20240502 01:55:57.166144 21601 ts_tablet_manager.cc:1875] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:57.183055 21600 raft_consensus.cc:2229] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:57.184185 21600 raft_consensus.cc:2258] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:57.200762 21600 ts_tablet_manager.cc:1875] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:57.222664 21601 ts_tablet_manager.cc:1888] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240502 01:55:57.222885 21601 log.cc:1196] T 6150a46aa8ca4d9bbd16c141ad2f1a3c P 6b9f9aaa1c86420190af75270914449a: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal/wals/6150a46aa8ca4d9bbd16c141ad2f1a3c
I20240502 01:55:57.223793 16068 catalog_manager.cc:4830] TS 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121): tablet 6150a46aa8ca4d9bbd16c141ad2f1a3c (table rebalance_test_table_3 [id=06d289343bab4b6997e72b3f3c3bdec8]) successfully deleted
I20240502 01:55:57.224429 21601 tablet_replica.cc:330] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a: stopping tablet replica
I20240502 01:55:57.224843 21601 raft_consensus.cc:2229] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:57.225265 21601 raft_consensus.cc:2258] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:57.242138 21601 ts_tablet_manager.cc:1875] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:57.244045 21600 ts_tablet_manager.cc:1888] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 01:55:57.248363 21601 ts_tablet_manager.cc:1888] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 01:55:57.259855 21600 log.cc:1196] T 3a4e2e002a7d441398f3da0ec650b8d2 P 0982e15b4c74425cad8d7d642eb63316: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-7/wal/wals/3a4e2e002a7d441398f3da0ec650b8d2
I20240502 01:55:57.267287 16486 consensus_queue.cc:237] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } }
I20240502 01:55:57.270994 16068 catalog_manager.cc:4830] TS 0982e15b4c74425cad8d7d642eb63316 (127.31.161.200:38559): tablet 3a4e2e002a7d441398f3da0ec650b8d2 (table rebalance_test_table_0 [id=ae3c0515a85746b591b7fd7a772c9884]) successfully deleted
I20240502 01:55:57.272630 18780 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:57.273474 21514 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:57.278544 16225 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:57.280211 21541 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:57.283185 21601 log.cc:1196] T f53b466fba6a4655a63a559b0ea56f77 P 6b9f9aaa1c86420190af75270914449a: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal/wals/f53b466fba6a4655a63a559b0ea56f77
I20240502 01:55:57.283986 16068 catalog_manager.cc:4830] TS 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121): tablet f53b466fba6a4655a63a559b0ea56f77 (table rebalance_test_table_1 [id=55b8533c6ab74c1f8b95e663c2bc47fa]) successfully deleted
I20240502 01:55:57.284528 21601 tablet_replica.cc:330] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a: stopping tablet replica
I20240502 01:55:57.284829 21601 raft_consensus.cc:2229] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:57.287194 21601 raft_consensus.cc:2258] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:57.288811 21601 ts_tablet_manager.cc:1875] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:57.291399 18913 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:57.292169 21541 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:57.296087 21514 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:57.301383 18913 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 0982e15b4c74425cad8d7d642eb63316 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:57.306998 17012 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:57.307777 21568 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:57.308173 21601 ts_tablet_manager.cc:1888] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240502 01:55:57.308287 21601 log.cc:1196] T a8f7e3c8b6d6434db2a99d7139599a5d P 6b9f9aaa1c86420190af75270914449a: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-5/wal/wals/a8f7e3c8b6d6434db2a99d7139599a5d
I20240502 01:55:57.308982 16068 catalog_manager.cc:4830] TS 6b9f9aaa1c86420190af75270914449a (127.31.161.198:38121): tablet a8f7e3c8b6d6434db2a99d7139599a5d (table rebalance_test_table_5 [id=33233b353a0644f9a68f71c6a04c7fd3]) successfully deleted
I20240502 01:55:57.309553 16620 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:57.311115 17012 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:57.313755 21568 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:57.319424 16076 catalog_manager.cc:5484] T 18e676bcd8b04f18a6814d6ebbd1b488 P a6575ebf978a4631a13ba5c243c3c4f9 reported cstate change: config changed from index 12 to 13, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New cstate: current_term: 1 leader_uuid: "39059854b64045698ded1ba4e8d47304" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:57.330090 16620 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 703f6f043c6041a4b629933690580367 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:57.340647 16067 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 18e676bcd8b04f18a6814d6ebbd1b488 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 01:55:57.343159 16883 raft_consensus.cc:1261] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Refusing update from remote peer 39059854b64045698ded1ba4e8d47304: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 01:55:57.344385 18780 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P aa7ec9abef1747e5a9e2404cee265c20 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:57.344939 16225 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P cdd4d50250184223992325c86cb6c68e [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:57.349781 16733 tablet_service.cc:1514] Processing DeleteTablet for tablet 18e676bcd8b04f18a6814d6ebbd1b488 with delete_type TABLET_DATA_TOMBSTONED (TS ed2e7e4188d0400b8689e28bbf5096ad not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:47194
I20240502 01:55:57.350545 21546 tablet_replica.cc:330] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad: stopping tablet replica
I20240502 01:55:57.350775 21546 raft_consensus.cc:2229] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 01:55:57.350920 21568 consensus_queue.cc:1035] T 18e676bcd8b04f18a6814d6ebbd1b488 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 01:55:57.351245 21546 raft_consensus.cc:2258] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 01:55:57.352612 16883 raft_consensus.cc:2932] T 18e676bcd8b04f18a6814d6ebbd1b488 P 6b9f9aaa1c86420190af75270914449a [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "0982e15b4c74425cad8d7d642eb63316" member_type: VOTER last_known_addr { host: "127.31.161.200" port: 38559 } attrs { promote: false } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20" member_type: VOTER last_known_addr { host: "127.31.161.202" port: 43977 } attrs { promote: false } } peers { permanent_uuid: "703f6f043c6041a4b629933690580367" member_type: VOTER last_known_addr { host: "127.31.161.196" port: 40425 } attrs { promote: false } } }
I20240502 01:55:57.356446 21546 ts_tablet_manager.cc:1875] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 01:55:57.365694 21546 ts_tablet_manager.cc:1888] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240502 01:55:57.365903 21546 log.cc:1196] T 18e676bcd8b04f18a6814d6ebbd1b488 P ed2e7e4188d0400b8689e28bbf5096ad: Deleting WAL directory at /tmp/dist-test-taskMchxvY/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714614747638897-32391-0/raft_consensus-itest-cluster/ts-4/wal/wals/18e676bcd8b04f18a6814d6ebbd1b488
I20240502 01:55:57.366806 16067 catalog_manager.cc:4830] TS ed2e7e4188d0400b8689e28bbf5096ad (127.31.161.197:46825): tablet 18e676bcd8b04f18a6814d6ebbd1b488 (table rebalance_test_table_2 [id=29f792777d054c50b859afc579ad1882]) successfully deleted
I20240502 01:55:57.398051 16489 consensus_queue.cc:237] T 27ce7f50b71e4f478cace3bde6f5e1a3 P 39059854b64045698ded1ba4e8d47304 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cdd4d50250184223992325c86cb6c68e" member_type: VOTER last_known_addr { host: "127.31.161.193" port: 41953 } } peers { permanent_uuid: "dc9fcb8d12cb4ab28ec821c9473adeeb" member_type: VOTER last_known_addr { host: "127.31.161.194" port: 38133 } } peers { permanent_uuid: "39059854b64045698ded1ba4e8d47304" member_type: VOTER last_known_addr { host: "127.31.161.195" port: 35693 } } peers { permanent_uuid: "6b9f9aaa1c86420190af75270914449a" member_type: VOTER last_known_addr { host: "127.31.161.198" port: 38121 } } peers { permanent_uuid: "ed2e7e4188d0400b8689e28bbf5096ad" member_type: VOTER last_known_addr { host: "127.31.161.197" port: 46825 } } peers { permanent_uuid: "a6575ebf978a4631a13ba5c243c3c4f9" member_type: VOTER last_known_addr { host: "127.31.161.199" port: 46837 } } peers { permanent_uuid: "a7c4573db86a415285d274ca57dde659" member_type: VOTER last_known_addr { host: "127.31.161.201" port: 44737 } attrs { promote: false replace: true } } peers { permanent_uuid: "aa7ec9abef1747e5a9e2404cee265c20"